Mozilla的编译系统是如何工作的

这篇翻译不完整。请帮忙从英语翻译这篇文章

本文档写给需要对Mozilla 编译系统进行修改的开发者,内容包括解释编译系统的基本概念和技术,以及如何完成一些基本的任务,例如编译各个模块及jar包。

对于大多数的开发者,使用mach build命令来编译源码树就足够了,本文是解释编译系统如何工作的。

注意:对于只是关心如何编译Mozilla软件的开发者来说,请参考Build Documentation.

Phases

当你输入mach build编译源码树时,编译系统做了一下3个主要编译阶段:

  1. 系统检测和验证
  2. 编译后台的准备
  3. 编译后台的调用

阶段1: configure

步骤1围绕configure脚本进行。configure脚是bash shell脚本,根据configure.in文件生成。使用GNU M4宏处理器编写,用Autoconf 2.13处理生成最终的脚本。你不需要担心如何获取configure文件,编译系统会为你做这个事情。

configure的主要工作是决定系统和编译器的特征,应用传入的选项,验证各项编译信息。configure的结果主要是在目标目录生成可执行的config.status文件。configure还会生成一些其他的文件(例如autoconf.mk)。当然,从系统层面来说最重要的是config.status文件。

config.status对于用过Autoconf的人会很熟悉,Mozilla的config.status与其他Autoconf生成的config.status不同,它是用Python写的!configure是shell脚本,而config.statusPython脚本。

Python 2在Mozilla的编译系统中很常见。如果需要在编译系统中写代码,我们写在Python脚本中而不是修改makefile文件。

config.status 包括2部分:

  • 表达configure输出结果的数据结构
  • 准备、配置、生成合适的编译后台的命令行接口。编译后台文件仅仅是一个用来生成编译树的工具,类似GNU Make 或者Tup

These data structures describe the current state of the system and what the existing build configuration looks like. For example, a data structure defines which compiler to use, how to invoke it, which application features are enabled, and so on. You are encouraged to open config.status to have a look!

After we have emitted a config.status file, we proceed to Phase 2.

阶段2:编译后台准备和编译定义

一旦configure确定了当前编译配置信息,我们需要将此配置应用到源码树以进行实际的编译。

config.status在生成后将会被立即执行,其任务是告诉编译工具如何生成编译树,为此config.status需要首先扫描编译系统定义。

编译系统定义由源码树中的许多moz.build文件构成。大致上,每个文件夹或一组相关的文件夹对应一个moz.build文件。每个moz.build文件定义了它这部分的编译配置如何进行工作。例如,它说我需要把某些C++文件编译或需要在某些文件夹中查找更多的信息。config.status从主moz.build开始,递归查找所有引用的文件和目录。随着moz.build文件被读取,描述整个编译系统定义的数据结构被激发。这些数据结构接着被编译后台生成器读取,然后被转换成文件、函数调用等等。在使用make后台的情况下,生成器输出makefile文件。

When config.status runs, you'll see the following output:

Reticulating splines...
Finished reading 1096 moz.build files into 1276 descriptors in 2.40s
Backend executed in 2.39s
2188 total backend files. 0 created; 1 updated; 2187 unchanged
Total wall time: 5.03s; CPU time: 3.79s; Efficiency: 75%

What this is saying is that a total of 1,096 moz.build files were read. Altogether, 1,276 data structures describing the build configuration were derived from them. It took 2.40s wall time just to read these files and produce the data structures. The 1,276 data structures were fed into the build backend, which then determined it had to manage 2,188 files derived from those data structures. Most of the files already existed and didn't need to be changed. However, one was updated as a result of the new configuration. The whole process took 5.03s. Of this, only 3.79s were in CPU time. This means we spent roughly 25% of the time waiting on I/O.

步骤3:编译后台的调用

当大多数人想到编译系统时,他们想到的就是步骤3。这是我们使用源码树中的所有代码生成Firefox二进制支持文件或其他应用的阶段。步骤3使用步骤2生成的结果进行实际工作。自从Mozilla诞生以来,这一步骤一直使用make工具,读取makefile文件进行工作。当然,随着moz.build文件的转换,你或许最终能看到非make后台,例如Tup或Visual Studio。

当编译源码树时,大多数的时间消耗在步骤3。在这一步,头文件被装配、C++文件被编译、文件被预处理等等。

Recursive Make Backend

The recursive make backend is the tried-and-true backend used to build the tree. It's what's been used since the dawn of Mozilla. Essentially, there are makefiles in each directory. make starts processing the makefile in the root directory and then recursively descends into child directories until it's done. But there's more to the process than that.

The recursive make backend divides the source tree into tiers. A tier is a grouping of related directories containing makefiles of their own. For example, there is a tier for the Netscape Portable Runtime (nspr), one for the JavaScript engine, one for the core Gecko platform, one for the XUL app being built, and so on.

The main moz.build file defines the tiers and directories in the tiers. In reality, the main moz.build files include other moz.build files, such as /toolkit/toolkit.mozbuild, which define the tiers. They do this via the add_tier_dir() function.

At build time, the tiers are traversed in the order they are defined. Typically, the traversal order looks something like base, nspr, nssjs, platform, app.

Each tier consists of three sub-tiers: export, libs, and tools. These sub-tiers roughly correspond to the actions of pre-build, main-build, and post-build. This naming, however, can be misleading because all three sub-tiers are part of the build:

  • export is used to do things like copy headers into place.
  • libs is reserved for most of the work, like compiling C++ source files.
  • tools is used to install tests and other support tools.

When make is invoked, it starts at the export sub-tier of the first tier, and traverses all the directories in that tier. Then, it does the same thing for the libs sub-tier and, subsequently, the tools sub-tier. It then moves on to the next tier and continues until no tiers remain.

To view information about the tiers, you can execute the following special make targets:

Command Effect
make echo-tiers Show the final list of tiers.
make echo-dirs Show the list of non-static source directories to iterate over, as determined by the tier list.
make echo-variable-STATIC_DIRS Show the list of static source directories to iterate over, as determined by the tier list.

moz.build Files

moz.build files are how each part of the source tree defines how it is integrated with the build system. Think of each moz.build file as a data structure telling the build system what to do.

During build backend generation, all moz.build files relevant to the current build configuration are read and converted into files and actions used to build the tree (such as makefiles). In this section, we'll talk about how moz.build files actually work.

An individual moz.build file is actually a Python script. However, they are unlike most Python scripts. The execution environment is strictly controlled, so moz.build files can only perform a limited set of operations. moz.build files are limited to performing the following actions:

  1. Calling functions that are explicitly made available to the moz.build environment.
  2. Assigning to a well-defined set of variables whose name is UPPERCASE.
  3. Creating new variables whose name is not UPPERCASE (this includes defining functions).

moz.build files cannot do the following:

  • Import modules.
  • Open files.
  • Use the print statement or function.
  • Reference many of Python's built-in or global functions (they are not made available to the execution environment).

The most important actions of moz.build files are #1 and #2 from the above list. These are how the execution of a moz.build file tells the build system what to do. For example, you can assign to the DIRS list to define which directories to traverse into looking for additional moz.build files.

The output of the execution of an individual moz.build file is a Python dictionary. This dictionary contains the UPPERCASE variables directly assigned to and the special variables indirectly assigned to by calling functions exported to the execution environment. This is what we were referring to when we said you can think of moz.build files as data structures.

moz.build UPPERCASE Variables and Functions

The set of special symbols available to moz.build files is centrally defined and is under the purview of the build configuration module. To view the variables and functions available in your checkout of the tree, run the following:

mach mozbuild-reference

Or, you can view the raw file at /python/mozbuild/mozbuild/frontend/context.py.

How moz.build Processing Works

For most developers, knowing that moz.build files are Python scripts that are executed and emit Python dictionaries describing the build configuration is enough. If you insist on knowing more, this section is for you.

All the code for reading moz.build files lives under /python/mozbuild/mozbuild/frontend/. mozbuild is the name of our Python package that contains most of the code for defining how the build system works. moz.build files and mozbuild are different, so be careful not to confuse the two.

sandbox.py contains code for a generic Python sandbox. This code is used to restrict the environment moz.build files are executed under.

reader.py contains the code that defines the moz.build sandbox (the MozbuildSandbox class) and the code for traversing a tree of moz.build files (the BuildReader class) by following DIRS and TIERS variables. A BuildReader is instantiated with a configuration, is told to read the source tree, and then emits a stream of MozbuildSandbox instances corresponding to the executed moz.build files.

The MozbuildSandbox stream produced by the BuildReader is typically fed into the TreeMetadataEmitter class from emitter.py. The role of TreeMetadataEmitter is to convert the low-level MozbuildSandbox dictionaries into higher-level function-specific data structures. These data structures are the classes defined in data.py. Each class defines a specific aspect of the build system, such as directories to traverse, C++ files to compile, and so on. The TreeMetadataEmitter output is a stream of instances of these classes.

The build system stream describing class instances emitted from TreeMetadataEmitter is then fed into a build backend. A build backend is simply an instance of a child class of a BuildBackend from base.py (in the mozbuild.backend package now, not mozbuild.frontend). The child class implements methods for processing individual class instances as well as common hook points, such as processing has finished. See recursivemake.py for an implementation of a BuildBackend.

Although we call the base class BuildBackend, the class doesn't need to be focused with building at all. If you wanted to create a consumer that performed a line count of all C++ files or generated a Clang compilation database, for example, this would be an acceptable use of a BuildBackend.

Technically, we don't need to feed the TreeMetadataEmitter output into a BuildBackend: it's possible to create your own consumer. However, a BuildBackend provides a common framework from which to author consumers. Along the same vein, you don't need to use TreeMetadataEmitter to consume MozbuildSandbox instances. Nor do you need to use BuildReader to traverse the moz.build files. This is just the default framework we've established for our build system.

Legacy Content

THE CONTENT BELOW IS CONSIDERED LEGACY. IT IS PRESERVED FOR HISTORICAL REASONS UNTIL THIS ENTIRE PAGE IS REWRITTEN.

Makefile basics

Makefiles can be quite complicated, but Mozilla provides a number of built-in rules that should enable most makefiles to be simpler. Complete documentation for make is beyond the scope of this document but is available here.

One concept you will need be familiar with is variables in make. Variables are defined by the syntax VARIABLE = VALUE, and the value of a variable is referenced by writing $(VARIABLE). All variables are strings.

All Makefile.in files in Mozilla have the same basic format:

DEPTH           = ../../../..
topsrcdir       = @top_srcdir@
srcdir          = @srcdir@
VPATH           = @srcdir@

include $(DEPTH)/config/autoconf.mk
# ... Main body of makefile goes here ...

include $(topsrcdir)/config/rules.mk

# ... Additional rules go here ...
  • The DEPTH variable should be set to the relative path from your Makefile.in to the top-level Mozilla directory.
  • topsrcdir is substituted in by configure and points to the top-level Mozilla directory.
  • srcdir is also substituted in by configure and points to the source directory for the current directory. In source tree builds, this will simply point to "." (the current directory).
  • VPATH is a list of directories where make will look for prerequisites (i.e., source files).

One other frequently used variable not specific to a particular build target is DIRS. DIRS is a list of subdirectories of the current directory to recursively build in. Subdirectories are traversed after their parent directories. For example, you could have:

DIRS = \
  public \
  resources \
  src \
  $(NULL)

This example demonstrates another concept called continuation lines. A backslash as the last character on a line allows the variable definition to be continued on the next line. The extra whitespace is compressed. The terminating $(NULL) is a method for consistency; it allows you to add and remove lines without worrying about whether the last line has an ending backslash or not.

Makefile examples

Building libraries

There are three main types of libraries that are built in Mozilla:

  • Components are shared libraries (except in static builds), which are installed to dist/bin/components. Components are not linked against by any other library.
  • Non-component shared libraries include libraries such as libxpcom and libmozjs. These libraries are installed to dist/bin and are linked against. You will probably not need to create a new library of this type.
  • Static libraries are often used as intermediate steps to building a shared library, if there are source files from several directories that are part of the shared library. Static libraries may also be linked into an executable.

Non-component shared libraries

A non-component shared library is useful when there is common code that several components need to share and sharing it through XPCOM is not appropriate or not possible. As an example, below is a portion of the makefile for libmsgbaseutil, which is linked against by all of the main news components:

 DEPTH           = ../../..
 topsrcdir       = @top_srcdir@
 srcdir          = @srcdir@
 VPATH           = @srcdir@
 
 include $(DEPTH)/config/autoconf.mk
 
 MODULE          = msgbaseutil
 LIBRARY_NAME    = msgbaseutil
 EXPORT_LIBRARY = 1
 SHORT_LIBNAME   = msgbsutl

Notice that the only change from the component example above is that IS_COMPONENT is not set. When this is not set, a shared library will be created and installed to dist/bin.

Static libraries

As mentioned above, static libraries are most commonly used as intermediate steps to building a larger library (usually a component). This lets you spread out the source files in multiple subdirectories. Static libraries may also be linked into an executable. As an example, below is a portion of the makefile from layout/base/src:

 DEPTH           = ../../..
 topsrcdir       = @top_srcdir@
 srcdir          = @srcdir@
 VPATH           = @srcdir@
 
 include $(DEPTH)/config/autoconf.mk
 
 MODULE          = layout
 LIBRARY_NAME    = gkbase_s
 
 # REQUIRES and CPPSRCS omitted here for brevity #
 
 # we don't want the shared lib, but we want to force the creation of a static lib.
 FORCE_STATIC_LIB = 1
 
 include $(topsrcdir)/config/rules.mk

The key here is setting FORCE_STATIC_LIB = 1. This creates libgkbase_s.a (on UNIX) and gkbase_s.lib on Windows and copies it to dist/lib. Now, let's take a look at how to link several static libraries together to create a component:

 DEPTH           = ../..
 topsrcdir       = @top_srcdir@
 srcdir          = @srcdir@
 VPATH           = @srcdir@
 
 include $(DEPTH)/config/autoconf.mk
 
 MODULE          = layout
 LIBRARY_NAME    = gklayout
 EXPORT_LIBRARY = 1
 IS_COMPONENT    = 1
 MODULE_NAME     = nsLayoutModule
 
 CPPSRCS         = \
                 nsLayoutModule.cpp \
                 $(NULL)
 
 SHARED_LIBRARY_LIBS = \
                 $(DIST)/lib/$(LIB_PREFIX)gkhtmlbase_s.$(LIB_SUFFIX) \
                 $(DIST)/lib/$(LIB_PREFIX)gkhtmldoc_s.$(LIB_SUFFIX) \
                 $(DIST)/lib/$(LIB_PREFIX)gkhtmlforms_s.$(LIB_SUFFIX) \
                 $(DIST)/lib/$(LIB_PREFIX)gkhtmlstyle_s.$(LIB_SUFFIX) \
                 $(DIST)/lib/$(LIB_PREFIX)gkhtmltable_s.$(LIB_SUFFIX) \
                 $(DIST)/lib/$(LIB_PREFIX)gkxulbase_s.$(LIB_SUFFIX) \
                 $(DIST)/lib/$(LIB_PREFIX)gkbase_s.$(LIB_SUFFIX) \
                 $(DIST)/lib/$(LIB_PREFIX)gkconshared_s.$(LIB_SUFFIX) \
                 $(DIST)/lib/$(LIB_PREFIX)gkxultree_s.$(LIB_SUFFIX) \
                 $(DIST)/lib/$(LIB_PREFIX)gkxulgrid_s.$(LIB_SUFFIX) \
                 $(NULL)
 
 include $(topsrcdir)/config/rules.mk

SHARED_LIBRARY_LIBS is set to a list of static libraries, which should be linked into this shared library. Note the use of LIB_PREFIX and LIB_SUFFIX to make this work on all platforms.

Building jar files

jar files are used for packaging chrome files (XUL, JavaScript, and CSS). For more information on jar packaging, see this document. Here, we will only cover how to set up a makefile to package jars. Below is an example:

 DEPTH           = ../../../..
 topsrcdir       = @top_srcdir@
 srcdir          = @srcdir@
 VPATH           = @srcdir@
 
 include $(DEPTH)/config/autoconf.mk
 
 include $(topsrcdir)/config/rules.mk

As you can see, there are no extra variables to define. If a jar.mn file exists in the same directory as this Makefile.in, it will automatically be processed. Although the common practice is to have a resources directory that contains the jar.mn and chrome files, you may also put a jar.mn file in a directory that creates a library, in which case it will be processed.

See the glossary of makefile variables for information about specific variables and how to use them.

Original Document Information

  • Author: Brian Ryner
  • Copyright Information: Portions of this content are © 1998–2006 by individual mozilla.org contributors; content available under a Creative Commons license

文档标签和贡献者

此页面的贡献者: Lingchar
最后编辑者: Lingchar,