=================
mozbuild.frontend
=================

The mozbuild.frontend package is of sufficient importance and complexity
to warrant its own README file. If you are looking for documentation on
how the build system gets started, you've come to the right place.

Overview
========

Tree metadata (including the build system) is defined by a collection of
files in the source tree called *mozbuild* files. These typically are files
named *moz.build*. But, the actual name can vary.

Each *mozbuild* file defines basic metadata about the part of the tree
(typically directory scope) it resides in. This includes build system
configuration, such as the list of C++ files to compile or headers to install
or libraries to link together.

*mozbuild* files are actually Python scripts. However, their execution
is governed by special rules. This will be explained later.

Once a *mozbuild* file has executed, it is converted into a set of static
data structures.

The set of all data structures from all relevant *mozbuild* files
constitute all of the metadata from the tree.

How *mozbuild* Files Work
=========================

As stated above, *mozbuild* files are actually Python scripts. However,
their behavior is very different from what you would expect if you executed
the file using the standard Python interpreter from the command line.

There are two properties that make execution of *mozbuild* files special:

1. They are evaluated in a sandbox which exposes a limited subset of Python
2. There is a special set of global variables which hold the output from
   execution.

The limited subset of Python is actually an extremely limited subset.
Only a few built-ins are exposed. These include *True*, *False*, and
*None*. Global functions like *import*, *print*, and *open* aren't defined.
Without these, *mozbuild* files can do very little. This is by design.

The side-effects of the execution of a *mozbuild* file are used to define
the build configuration. Specifically, variables set during the execution
of a *mozbuild* file are examined and their values are used to populate
data structures.

The enforced convention is that all UPPERCASE names inside a sandbox are
reserved and it is the value of these variables post-execution that is
examined. Furthermore, the set of allowed UPPERCASE variable names and
their types is statically defined. If you attempt to reference or assign
to an UPPERCASE variable name that isn't known to the build system or
attempt to assign a value of the wrong type (e.g. a string when it wants a
list), an error will be raised during execution of the *mozbuild* file.
This strictness is to ensure that assignment to all UPPERCASE variables
actually does something. If things weren't this way, *mozbuild* files
might think they were doing something but in reality wouldn't be. We don't
want to create false promises, so we validate behavior strictly.

If a variable is not UPPERCASE, you can do anything you want with it,
provided it isn't a function or other built-in. In other words, normal
Python rules apply.

All of the logic for loading and evaluating *mozbuild* files is in the
*reader* module. Of specific interest is the *MozbuildSandbox* class. The
*BuildReader* class is also important, as it is in charge of
instantiating *MozbuildSandbox* instances and traversing a tree of linked
*mozbuild* files. Unless you are a core component of the build system,
*BuildReader* is probably the only class you care about in this module.

The set of variables and functions *exported* to the sandbox is defined by
the *sandbox_symbols* module. These data structures are actually used to
populate MozbuildSandbox instances. And, there are tests to ensure that the
sandbox doesn't add new symbols without those symbols being added to the
module. And, since the module contains documentation, this ensures the
documentation is up to date (at least in terms of symbol membership).

How Sandboxes are Converted into Data Structures
================================================

The output of a *mozbuild* file execution is essentially a dict of all
the special UPPERCASE variables populated during its execution. While these
dicts are data structures, they aren't the final data structures that
represent the build configuration.

We feed the *mozbuild* execution output (actually *reader.MozbuildSandbox*
instances) into a *TreeMetadataEmitter* class instance. This class is
defined in the *emitter* module. *TreeMetadataEmitter* converts the
*MozbuildSandbox* instances into instances of the *TreeMetadata*-derived
classes from the *data* module.

All the classes in the *data* module define a domain-specific
component of the tree metadata, including build configuration. File compilation
and IDL generation are separate classes, for example. The only thing these
classes have in common is that they inherit from *TreeMetadata*, which is
merely an abstract base class.

The set of all emitted *TreeMetadata* instances (converted from executed
*mozbuild* files) constitutes the aggregate tree metadata. This is the
the authoritative definition of the build system, etc and is what's used by
all downstream consumers, such as build backends. There is no monolithic
class or data structure. Instead, the tree metadata is modeled as a collection
of *TreeMetadata* instances.

There is no defined mapping between the number of
*MozbuildSandbox*/*moz.build* instances and *TreeMetadata* instances.
Some *mozbuild* files will emit only 1 *TreeMetadata* instance. Some
will emit 7. Some may even emit 0!

The purpose of this *emitter* layer between the raw *mozbuild* execution
result and *TreeMetadata* is to facilitate additional normalization and
verification of the output. There are multiple downstream consumers of
this data and there is common functionality shared between them. An
abstraction layer that provides high-level filtering is a useful feature.
Thus *TreeMetadataEmitter* exists.

Other Notes
===========

*reader.BuildReader* and *emitter.TreeMetadataEmitter* have a nice
stream-based API courtesy of generators. When you hook them up properly,
*TreeMetadata* instances can be consumed before all *mozbuild* files have
been read. This means that errors down the pipe can trigger before all
upstream tasks (such as executing and converting) are complete. This should
reduce the turnaround time in the event of errors. This likely translates to
a more rapid pace for implementing backends, which require lots of iterative
runs through the entire system.

Lots of code in this sub-module is applicable to other systems, not just
Mozilla's. However, some of the code is tightly coupled. If there is a will
to extract the generic bits for re-use in other projects, that can and should
be done.