parrot directory reorganization

i propose reorganizing the tree to improve clarity and cohesion.
here is an annotated picture of the current tree (extra info
<snip>ped as shown.)

. trunk
+---ast
+---blib
| +---lib
+---build_tools # move to tools/build/
+---charset # move to src/charset/
+---classes # move to src/classes/
+---compilers
| +---pge
| | +---PGE
| +---tge
| +---TGE
+---config
<snip config/ subdirs>
+---debian
+---docs
<snip docs/ subdirs>
+---dynclasses # move to src/dynclasses/
+---dynoplibs # move to src/dynoplibs/
+---editor # move to tools/editor/, get rid of
Makefile, convert it to perl
+---encodings # move to src/encodings/
+---examples
<snip examples/ subdirs>
+---imcc
| +---docs
| +---examples
| +---t # move to t/compilers/imcc/
<snip imcc/t/ subdirs>
+---include
| +---parrot
| +---oplib
+---io # move to src/io/
+---jit # move to src/jit/
<snip jit/ subdirs>
+---languages
<snip languages/ subdirs>
+---LICENSES
+---ops # move to src/ops/
+---pf # move to src/packfile/
+---runtime
<snip runtime/ subdirs>
+---src
+---t
| +---compilers
| | +---pge
| | | +---p5regexp
| | | +---p6rules
| | +---tge
| +---doc
| +---dynclass
| +---examples
| +---library
| +---native_pbc
| +---op
| +---perl
| +---pmc
| +---run
| +---src
| +---stress
| +---tools
+---tools
| +---dev
| +---docs
+---types # move to src/types/
+---util # move to tools/util/

below is the modified tree, which groups related directories,
and reduces the number of top-level directories from 28 to 15.

. trunk
+---ast
+---blib
| +---lib
+---compilers
| +---pge
| | +---PGE
| +---tge
| +---TGE
+---config
| <snip config/ subdirs>
+---debian
+---docs
<snip docs/ subdirs>
+---examples
| <snip examples/ subdirs>
+---imcc
| +---docs
| +---examples
+---include
| +---parrot
| +---oplib
+---languages
| <snip languages/ subdirs>
+---LICENSES
+---runtime
| <snip runtime/ subdirs>
+---src
| +---charset # moved from charset/
| +---classes # moved from classes/
| +---dynclasses # moved from dynclasses/
| +---dynoplibs # moved from dynoplibs/
| +---encodings # moved from encodings/
| +---io # moved from io/
| +---jit # moved from jit/
| | <snip jit/ subdirs>
| +---ops # moved from ops/
| +---pf # moved from pf/
| +---types # moved from types/
+---t
| +---compilers
| | +---imcc # moved from imcc/t/
| | | <snip t/compilers/imcc/ subdirs>
| | +---pge
| | | +---p5regexp
| | | +---p6rules
| | +---tge
| +---doc
| +---dynclass
| +---examples
| +---library
| +---native_pbc
| +---op
| +---perl
| +---pmc
| +---run
| +---src
| +---stress
| +---tools
+---tools
| +---build_tools # moved from build/
| +---dev
| +---docs
| +---editor # moved from editor/, no Makefile,
converted to perl
| +---util # moved from util/

this work can and should be done one piece at a time to prevent
widespread panic. if this proposal is accepted, i will create
todo tasks for each unit of work. then, they can be assigned one
at a time, until the task is complete. of course, a realclean /
configure / make test cycle should be done after each change to
ensure it has gone smoothly.

thoughts?
~jerry

I'm just curious, I don't mean to criticize your ideas. Just, give
some "why"s for the "should"s.

Why must all perl scripts be grouped under one directory?

Why must all C files be grouped under one directory?

Why can't we use make for this? Make handles modification times and
rebuilds on demand based on dependencies. imc.vim (and a few others
IIRC) are built from dependencies, and if the dependencies haven't
been changed, they don't need to be rebuilt. That sounds like make's
job to me.

Luke

perhaps my language was too imprecise here. let me try to clarify.
there are multiple directories off the root that contain tools and
utilities (most, if not all of which, are written in perl.) some are
used during development, some during build, and some during test.
grouping these tools and utilities together will give parrot
developers (and users) greater ability to infer the purpose of the
files based on their directory locations.

it requires too much knowledge of parrot in order to know that
development tools belong under tools/dev/, testing tools like the
smoke server belong under utils/, and build tools can be found under
build_tools/. i believe moving these directories under one parent
(tools) would flatten the learning curve.

again, perhaps i was imprecise--it's not a breakdown by language type
i'm looking for. in fact, the files in ops/, dynclasses/ and classes/
aren't even C files -- C files are generated from them. but, they are
logically source files. keeping all source files under one parent
would increase clarity and flatten the learning curve by keeping like
things together.

perhaps you are right here. there is a makefile target ('tags') which
calls the editor makefile. i don't have or use ctags, so i forgot
there was even a target for making tags. the tags target is not run as
part of the default make target, so only if a developer calls 'make
tags' will the editor makefile run. the editor syntax files are
regenerated if any ops files change, so there is indeed a dependency
there.

however, the syntax files are not installed, even if 'make tags' is
run. you must 'cd editor && make vim-install' (or 'kate-install') if
you want to actually update your editor. so, really, i don't see the
purpose of having this as a make step, since it doesn't finish the job
(and shouldn't! make shouldn't just install things whereever it wants
to on your system without you telling it to, like typing 'make
install'.)

anyway, i'll leave decisions on removing the makefile from editor/ up
to chip and leo, but my feelings are that it doesn't belong as part of
the make cycle, and is better off as a seperate utility, like
regenerating the MANIFEST.SKIP file (also a seperate utility.)

thanks for the feedback. hopefully i have better conveyed the reasons
behind my proposal.
~jerry

Reorganization would be good at this point, but should be undertaken
cautiously so as to minimize unpleasantness.

* I can't find a rationale for putting both 'imcc' and 'compilers' at
the top level. (Though I see a note that you intended imcc to be
under compilers; if so, I think that's good.)

* Should pasm be considered a compiler for source organization
purposes? I suppose in theory it should: There's more than one
assembler syntax for most CPUs. And AFAICT, the actual pasm
implementation is intermixed with the PIR implementation, so if imcc
goes into compilers, it'll take pasm with it.

* The directory name 'src' is a bit fuzzy to me, if 'imcc' and
'compilers' aren't part of it. Perhaps 'vm' for the core
machine...? That would include infrastructure PMCs.

Lots more interesting ideas, I hope.

(Jerry's diagrams follow.)

--
Chip Salzenberg <ch...@pobox.com>

Sounds reasonable.

I've found that reorganizing source trees tends to expose lots of build
dependency problems so it's probably worth doing for that reason alone.
However, it should be done with some caution and lots of warning for
those that have uncommitted changes.

I disagree. A Perl script won't do any better of a job and will be more
complex then that rather simple makefile.

Does it make sense for the tests to even refer to imcc? IMHO Parrot
ability to digest PIR is whats really being tested. However unlikely,
it's worth keeping in ind that Parrots internal PIR compiler may be
replaced someday.

[SNIP]

I'd suggest that you consider making the change in two phases. With all
of the non-"move it under /src" changes first as it will cause less
disruption in the tree.

Cheers,

-J

--

I mildly agree with Joshua about Makefiles. More to the point, I
don't like changing directory tree and build tech at the same time.
They're separate changes.

Oh, one other thing for the renaming game:

IMC vs. PIR
Two names enter
One name leaves

/me giggles
--
Chip Salzenberg <ch...@pobox.com>

Agreed. I find this confusing sometimes, not being intimately familiar
with all of parrot.

<snip />

> thoughts?

I'd really like to see imcc/docs/ get moved to docs/imcc/ while we're
moving things around. I know that some people consider imcc to be
rather separate from parrot, but as a compiler writer, I see it as a
core feature. I always seem to forget that there are docs in
imcc/docs/ and end up grepping through docs/ instead (missing valuable
information).

--
matt diephouse
http://matt.diephouse.com

Yes. But first make sure that build_tools contains really all build
tools, which it doesn't.
tools/dev/cc_flags.pl is AFAIK used during build. OTOH
build_tools/list_unjitted.pl is just a tool.

Nope - why a perl script for a make job?

ast is another builtin parrot compiler BTW.

> +---imcc
> | +---docs

Should be merged into docs/

> ... of course, a realclean /

... on several different platforms.

> thoughts?
> ~jerry

leo

This doesn't really work for me. imcc/t* used to test some general
parsing features, code generation and register allocation. Therefore it
should run early in the tests.

What about this file organization and testing order:

t/imcc # 1. or 2.
/op # 2. or 1.
/pmc
/dynclass
...
# more tests

We could also get rid of the imcc prefix and move tests and test files
to more appropriate files and just delete some duplicates.

Also t/compilers is now split into 2 passes with different tests, which
looks strange to me.

leo

i'd like to see test failures occur as quickly as possible to make the
development cycle at least a little quicker (the test suite takes
almost 20mins on win32--however some platforms are *much* quicker.)
i'll take your info above into account when reordering the tests.

i'll take a closer look at these tests and think about what's
appropriate. expect questions, either here or on #parrot.

indeed, and it will be changed soon. it still may be in two or more
passes as the grouping of tests at the moment is by logical similarity
(e.g. compilers are grouped together,) but the ordering will be by
importance of catching functional failures. if you see the order looks
a little different than you like, please speak up, or just patch it :)
~jerry

if any tests are moved in the future, i'll make sure to pay close
attention to their order in the harness.
~jerry

i intended for the tests to be under t/compilers/, only because i have
previously been focusing on a test suite reorganization. that's almost
complete now, and has led me to these observations of the parrot
directory structure. i agree that imcc should be better incorporated
in the tree, and i think compilers/ is the best place for it. also,
ast/ (which leo mentioned) will be moved under compilers/ as well.

looks like you answered your own question there.

good idea. vm/ sounds good to me, but i'll wait for feedback on that
before making any changes. besides, src/ related changes will probably
happen last, anyway.

~jerry

~jerry

due to popular revolt, i withdraw my proposal to move editor/
directory or it's makefile. no reason to swim upstream--there's plenty
of other work to do here.

<snip issues which have been addressed in other messages>
~jerry