MOST OF THIS IS ANCIENT AND NO LONGER APPLIES. I WILL TRY AND MARK JUST PLAIN WRONG
SECTIONS WITH AN ASTERISK.

\section{Introduction}

This new build framework is intended to manage all phases on the construction of software libraries and
applications. Thus, it must interface with the version control subsystem, manage architecture customization and
interoperability, and construct libraries and executables efficiently. This structure must be portable to many
architectures, but we may assume some sophistication on the part of the systems administrator. Finally, we term this a
framework as it is intended to be extended, customized, and evolved by its users. This is the critical component that is
missing from all current solutions.

The controlling metaphor of this work is that of a pipeline. Each component in the pipeline is seen as a Transform, or
set-to-set mapping. A set of files is accepted as input and a second set is delivered as output. We put no a priori
semantic conditions on these sets. Eventually we expect to construct general graphs linking these transform objects,
however currently we are working with only linear arrays and DAGs. We also introduce an abstraction of a group of files,
the FileGroup. This allows sets to be implicitly defined, and to define sets which evolve in time.

This framework is intended to support SIDL builds, which will consist in part of generated code. Thus, we are forced to
abandon timestamps as a method for detecting file changes. Timestamps are replaced by checksums (md5 by default), even
for archives and other binary formats.

We choose Python as the implementation language for several reasons. It satisfies our portability criteria, as it runs
on every architecture which we believe will be supported and it relatively easy to port to those new architectures which
will appear. Furthermore, it has strong community support and can be expected to be a dynamic, living language for at
least a decade. It is object-oriented and intepreted, yet it shows good performance compared to other scripting language
competitors. Its dynamic compilation and loading capability is essential, and will allow a truly new development paradigm.

\section{Projects}

A {\em Project} is the fundamental unit of code in the build system. Projects can be retrieved and constructed using the
installer. They provide a set of client libraries, as well as a set of server, or implementation libraries. Each project
is uniquely identified by its Bitkeeper URI, e.g. bk://sidl.bkbits.net/Compiler. Dependecies can be expressed between
projects using the project URI, and these allow automatica installation of dependent packages.

A project is a rooted directory tree which possesses a {\tt make.py} file at the root. Typically, it also has a {\tt
sidl} directory containing all the SIDL files, a {\tt lib} directory for libraries, a {\tt bin} or {\tt driver}
directory for executables, and several client and server directories generated by the SIDL compiler.

\section{Makefiles}

\section{Simultaneous Builds}

We will first restrict ourselves to the case in which the builds take place in separate interpreter instances, rather
than independent threads within the same interpreter. If these two builds share the same BuildSystem project, then they
will also share the same argument dictionary parent. Several variables are updating during the build process, and we
must examine their interaction.
\begin{itemize}
  \item FileSets generated by SIDL compilations are cached. Therefore if the same project is referenced by both builds,
there is a race condition here. However, the order should not affect the outcome of either build.

  \item The 'target' variable in RDict will not be shared since each build will have its own RDict object.

  \item FIX: The SIDL compiler currently puts its mode and output information in the argument dictionary. These arguments
must be declared local in order to avoid interaction.

  \item FIX: Currently 'installedLanguages' is used to determine whether the language is available. This will be changed
to detect the using* module associated to it.

  \item FIX: If a new language generator is being installed along with another project, the generator could finish
before the project, resulting in an incompatbility for the 'clientLanguages'key. This will be fixed when clients are
automatically built for dpendencies.

  \item FIX: There is currently a race condition for configure variables. If a project which has not been configured is
used in simultaneous builds, one build may start the configure, while the other detects the log file and immediately
asks for a configure variable which is not yet set.

  \item There is an obvious race conitions in building two projects which have a dependence relation, which is expressed
in the 'installedprojects' and 'projectDependenceGraph' variables. Furthermore, the dependence graph will be
indeterminate until both builds finish. However, it will always be in a valid state.

  \item Python configure variables are stored, but they will be identical for the same Python. Thus, problems could
arise if two different versions of Python were used to do simultaneous builds.

  \item All variables controlling the build operation itself, such as 'noStackTrace', are declared local, and thus will
not interact.

  \item The 'checkpoint' argument could potentially conflict, Thus checkpoints should not be generated during
simultaneous builds right now.

\end{itemize}

If we are in the same process space, then the default argDB object from Base will also be shared. This presents a
problem for the 'target' variable. The temporary directory is named by process id, which on Linux is different for each
thread, but this may not be the case for some operating systems.

\section{*FileGroup}

We see three main methods for defining a group of files. The first is a straightforward list of files, embodied by a
Python list of strings. It may make sense to introduce an abstraction for the file object itself, but this does not mesh
with the Python supporting modules (os, and os.path), and therefore we start with a simpe string.

Second, we can imagine defining a set of files implicitly using a function. For instance, we might ask for all the files
with extension ``cc'' under a certain directory root. Currently, we have no facility for caching such results as it is
unclear how to determine when changes will occur. Thus such functions will have to be reevaluated for each
query. Therefore we provide the getFiles() method which returns the set as a Python list, which may then support several
such queries.

Lastly, FileGroup objects should be composable in a tree fashion. So that the set of files which a parent defines
includes the set defined by each of its children. The final set defined will contain the disjoint union of the files
produced by each method.

Some common file sets are defined by subclasses of FileGroup. The TreeFileGroup defines a set of files by including any
file beneath a given root which satify a given user test. The ExtensionFileGroup provides this test, which checks that
the file extension matches a user-supplied string.

\section{*Maker}

This base class is intended to support any object intended to be used within the build system, and thus has very few,
basic capabilities. It should manage temporary space. It also understands the execution of a shell command. Currently,
this is the most common operation in any build system. Ideally, we would like to interoperate at a library level with
the OS and all dependent packages, but that day still seems somewhat far away. Also, a custom error handler for the
function may be provided.

\section{Transform}

Each object in the build pipeline is intended to be a Transform. The transform itself is effected using the execute()
method. This maps the Transform.sources FileGroup into the Transform.products FileGroup.

We divide the hierarchy below Transform into components which produce sets satisfying a predicate, of which only
FileCompare is currently a representative, and components which execute a given action across the input FileGroup
(possibly supplemented by additional file sets), of which perhaps the most common is CompileFiles, the abstraction of
the compilation process.

Since processing of files is so intimately connected to the transformation operation, we also include a getObjectName()
method in Transform which aids in the naming of intermediate files. This naming scheme must be accessible to both the
predicates and the actions, and thus has been pushed up to the Transform itself.

\section{*FileCompare}

A common operation is to compare a set of target files to the input set, in order to generate the set of products. By
default, each target is compared to each source (using self.compare), and each source file satifying the relation is
added to the output set. If a target is nonexistent, we assume that it needs to be acted upon and add all sources to the
output. Likewise, a nonexistent source is automatically added to the output set.

Some common predicates included in the current package are OlderThan, NewerThan, and NewerThanLibraryObject, the last of
which deserves some explanation. The comparsion routine file gets the corresponding object name for the source file
using Transform.getObjectName(), which is then compared with the timestamp of the same object in the target library. If
the object does not exist in the library, we treat it as if the source file itself was absent (adding it to the output).

\section{*Action}

An Action is meant to embody the execution of a function over a FileGroup, producing another, specifcally a function
which has external results such as creation or deletion of files. Moreover, it has the connotation of an external
program in the role of function.

We have two execution modes, in which the program is executed on each file individually, or on the FileGroup as a
whole. Furthermore, we allow a filter to be interposed which selects a subset of the input. We could perhaps promote
this to an object itself. The program argument can be either an external program name or a Python function, and the flag
argument can be used to pass command line arguments, and an error handler for shell commands can be provided.

\section{*CompileFiles}

The base class for all compilation, CompileFiles, defines the main features which are customized for each compiler. It
is not intended to be used directly as a component. The target is an archive which will hold all the intermediate
objects produced from the set of source files. It accepts a compiler, compilerFlags, archiver, and archiverFlags. It
also propogates the allAtOnce flag used in Action, indicating whether each file should be compiled separately, and the
fileFliter to select source files. The CompileCFiles, compileCxxFiles, and compileSIDLFiles objects just provide default
values for these arguments.

\section{*Linking}

The LinkSharedLibrary components transforms archives into shared libraries. It uses the archiver to expand each library
into objects, which are then linked into a shared libary. The user can provide extraLibraries in order to complete the
link. In the same way, LinkExecutable will map a FileGroup of libraries into an executable (perhaps with a set of
extraLibraries).

\section{*BK Interface}

We provide a simple interface to BitKeeper so that generated source files which are also edited may be stored under
version control. BKEditFiles will edit the incoming file set, which may be augmented using the extraSouces argument in
the event that we know the generated file set composition but only want to execute when it changes. BKCloseFiles, which
generally takesa TreeFileGroup, has a slightly more complicated action on the input set:
\begin{\itemize}
  \item All files generated which are unknown to BK are put underversion control
  \item All unchanged files are unedited
  \item All changed files are left edited (we could perhaps force a checkin here)
\end{\itemize}

\section{*Target}

Target provides an entry point into the build system from the interpreter. Targets should be able to be composed,
although this feature has not yet been tested. Users will provide a Target name for execution, which may itself include
internal Targets (themselves accessible by the same method). A Target is itself a Transform, and thus can be thought of
as merely a map of one FileGroup to another.
