src/third_party/mozjs-45/build/docs/slow.rst - cobalt - Git at Google

 .. _slow:

 ============================
 Why the Build System is Slow
 ============================

 A common complaint about the build system is that it's slow. There are
 many reasons contributing to its slowness. We will attempt to document
 them here.

 First, it is important to distinguish between a :term:`clobber build`
 and an :term:`incremental build`. The reasons for why each are slow can
 be different.

 The build does a lot of work
 ============================

 It may not be obvious, but the main reason the build system is slow is
 because it does a lot of work! The source tree consists of a few
 thousand C++ files. On a modern machine, we spend over 120 minutes of CPU
 core time compiling files! So, if you are looking for the root cause of
 slow clobber builds, look at the sheer volume of C++ files in the tree.

 You don't have enough CPU cores and MHz
 =======================================

 The build should be CPU bound. If the build system maintainers are
 optimizing the build system perfectly, every CPU core in your machine
 should be 100% saturated during a build. While this isn't currently the
 case (keep reading below), generally speaking, the more CPU cores you
 have in your machine and the more total MHz in your machine, the better.

 **We highly recommend building with no fewer than 4 physical CPU
 cores.** Please note the *physical* in this sentence. Hyperthreaded
 cores (an Intel Core i7 will report 8 CPU cores but only 4 are physical
 for example) only yield at most a 1.25x speedup per core.

 We also recommend using the most modern CPU model possible. Haswell
 chips deliver much more performance per CPU cycle than say Sandy Bridge
 CPUs.

 This cause impacts both clobber and incremental builds.

 You are building with a slow I/O layer
 ======================================

 The build system can be I/O bound if your I/O layer is slow. Linking
 libxul on some platforms and build architectures can perform gigabytes
 of I/O.

 To minimize the impact of slow I/O on build performance, **we highly
 recommend building with an SSD.** Power users with enough memory may opt
 to build from a RAM disk. Mechanical disks should be avoided if at all
 possible.

 Some may dispute the importance of an SSD on build times. It is true
 that the beneficial impact of an SSD can be mitigated if your system has
 lots of memory and the build files stay in the page cache. However,
 operating system memory management is complicated. You don't really have
 control over what or when something is evicted from the page cache.
 Therefore, unless your machine is a dedicated build machine or you have
 more memory than is needed by everything running on your machine,
 chances are you'll run into page cache eviction and you I/O layer will
 impact build performance. That being said, an SSD certainly doesn't
 hurt build times. And, anyone who has used a machine with an SSD will
 tell you how great of an investment it is for performance all around the
 operating system. On top of that, some automated tests are I/O bound
 (like those touching SQLite databases), so an SSD will make tests
 faster.

 This cause impacts both clobber and incremental builds.

 You don't have enough memory
 ============================

 The build system allocates a lot of memory, especially when building
 many things in parallel. If you don't have enough free system memory,
 the build will cause swap activity, slowing down your system and the
 build. Even if you never get to the point of swapping, the build system
 performs a lot of I/O and having all accessed files in memory and the
 page cache can significantly reduce the influence of the I/O layer on
 the build system.

 **We recommend building with no less than 8 GB of system memory.** As
 always, the more memory you have, the better. For a bare bones machine
 doing nothing more than building the source tree, anything more than 16
 GB is likely entering the point of diminishing returns.

 This cause impacts both clobber and incremental builds.

 You are building on Windows
 ===========================

 New processes on Windows are about a magnitude slower to spawn than on
 UNIX-y systems such as Linux. This is because Windows has optimized new
 threads while the \*NIX platforms typically optimize new processes.
 Anyway, the build system spawns thousands of new processes during a
 build. Parts of the build that rely on rapid spawning of new processes
 are slow on Windows as a result. This is most pronounced when running
 *configure*. The configure file is a giant shell script and shell
 scripts rely heavily on new processes. This is why configure on Windows
 can run over a minute slower on Windows.

 Another reason Windows builds are slower is because Windows lacks proper
 symlink support. On systems that support symlinks, we can generate a
 file into a staging area then symlink it into the final directory very
 quickly. On Windows, we have to perform a full file copy. This incurs
 much more I/O. And if done poorly, can muck with file modification
 times, messing up build dependencies. As of the summer of 2013, the
 impact of symlinks is being mitigated through the use
 of an :term:`install manifest`.

 These issues impact both clobber and incremental builds.

 Recursive make traversal is slow
 ================================

 The build system has traditionally been built by employing recursive
 make. Recursive make involves make iterating through directories / make
 files sequentially and executing each in turn. This is inefficient for
 directories containing few targets/tasks because make could be *starved*
 for work when processing these directories. Any time make is starved,
 the build isn't using all available CPU cycles and the build is slower
 as a result.

 Work has started in bug 907365 to fix this issue by changing the way
 make traverses all the make files.

 The impact of slow recursive make traversal is mostly felt on
 incremental builds. Traditionally, most of the wall time during a
 no-op build is spent in make traversal.

 make is inefficient
 ===================

 Compared to modern build backends like Tup or Ninja, make is slow and
 inefficient. We can only make make so fast. At some point, we'll hit a
 performance plateau and will need to use a different tool to make builds
 faster.

 Please note that clobber and incremental builds are different. A clobber
 build with make will likely be as fast as a clobber build with e.g. Tup.
 However, Tup should vastly outperform make when it comes to incremental
 builds. Therefore, this issue is mostly seen when performing incremental
 builds.

 C++ header dependency hell
 ==========================

 Modifying a *.h* file can have significant impact on the build system.
 If you modify a *.h* that is used by 1000 C++ files, all of those 1000
 C++ files will be recompiled.

 Our code base has traditionally been sloppy managing the impact of
 changed headers on build performance. Bug 785103 tracks improving the
 situation.

 This issue mostly impacts the times of an :term:`incremental build`.

 A search/indexing service on your machine is running
 ====================================================

 Many operating systems have a background service that automatically
 indexes filesystem content to make searching faster. On Windows, you
 have the Windows Search Service. On OS X, you have Finder.

 These background services sometimes take a keen interest in the files
 being produced as part of the build. Since the build system produces
 hundreds of megabytes or even a few gigabytes of file data, you can
 imagine how much work this is to index! If this work is being performed
 while the build is running, your build will be slower.

 OS X's Finder is notorious for indexing when the build is running. And,
 it has a tendency to suck up a whole CPU core. This can make builds
 several minutes slower. If you build with ``mach`` and have the optional
 ``psutil`` package built (it requires Python development headers - see
 :ref:`python` for more) and Finder is running during a build, mach will
 print a warning at the end of the build, complete with instructions on
 how to fix it.
	.. _slow:

	============================
	Why the Build System is Slow
	============================

	A common complaint about the build system is that it's slow. There are
	many reasons contributing to its slowness. We will attempt to document
	them here.

	First, it is important to distinguish between a :term:`clobber build`
	and an :term:`incremental build`. The reasons for why each are slow can
	be different.

	The build does a lot of work
	============================

	It may not be obvious, but the main reason the build system is slow is
	because it does a lot of work! The source tree consists of a few
	thousand C++ files. On a modern machine, we spend over 120 minutes of CPU
	core time compiling files! So, if you are looking for the root cause of
	slow clobber builds, look at the sheer volume of C++ files in the tree.

	You don't have enough CPU cores and MHz
	=======================================

	The build should be CPU bound. If the build system maintainers are
	optimizing the build system perfectly, every CPU core in your machine
	should be 100% saturated during a build. While this isn't currently the
	case (keep reading below), generally speaking, the more CPU cores you
	have in your machine and the more total MHz in your machine, the better.

	**We highly recommend building with no fewer than 4 physical CPU
	cores.** Please note the physical in this sentence. Hyperthreaded
	cores (an Intel Core i7 will report 8 CPU cores but only 4 are physical
	for example) only yield at most a 1.25x speedup per core.

	We also recommend using the most modern CPU model possible. Haswell
	chips deliver much more performance per CPU cycle than say Sandy Bridge
	CPUs.

	This cause impacts both clobber and incremental builds.

	You are building with a slow I/O layer
	======================================

	The build system can be I/O bound if your I/O layer is slow. Linking
	libxul on some platforms and build architectures can perform gigabytes
	of I/O.

	To minimize the impact of slow I/O on build performance, **we highly
	recommend building with an SSD.** Power users with enough memory may opt
	to build from a RAM disk. Mechanical disks should be avoided if at all
	possible.

	Some may dispute the importance of an SSD on build times. It is true
	that the beneficial impact of an SSD can be mitigated if your system has
	lots of memory and the build files stay in the page cache. However,
	operating system memory management is complicated. You don't really have
	control over what or when something is evicted from the page cache.
	Therefore, unless your machine is a dedicated build machine or you have
	more memory than is needed by everything running on your machine,
	chances are you'll run into page cache eviction and you I/O layer will
	impact build performance. That being said, an SSD certainly doesn't
	hurt build times. And, anyone who has used a machine with an SSD will
	tell you how great of an investment it is for performance all around the
	operating system. On top of that, some automated tests are I/O bound
	(like those touching SQLite databases), so an SSD will make tests
	faster.

	This cause impacts both clobber and incremental builds.

	You don't have enough memory
	============================

	The build system allocates a lot of memory, especially when building
	many things in parallel. If you don't have enough free system memory,
	the build will cause swap activity, slowing down your system and the
	build. Even if you never get to the point of swapping, the build system
	performs a lot of I/O and having all accessed files in memory and the
	page cache can significantly reduce the influence of the I/O layer on
	the build system.

	We recommend building with no less than 8 GB of system memory. As
	always, the more memory you have, the better. For a bare bones machine
	doing nothing more than building the source tree, anything more than 16
	GB is likely entering the point of diminishing returns.

	This cause impacts both clobber and incremental builds.

	You are building on Windows
	===========================

	New processes on Windows are about a magnitude slower to spawn than on
	UNIX-y systems such as Linux. This is because Windows has optimized new
	threads while the \*NIX platforms typically optimize new processes.
	Anyway, the build system spawns thousands of new processes during a
	build. Parts of the build that rely on rapid spawning of new processes
	are slow on Windows as a result. This is most pronounced when running
	configure. The configure file is a giant shell script and shell
	scripts rely heavily on new processes. This is why configure on Windows
	can run over a minute slower on Windows.

	Another reason Windows builds are slower is because Windows lacks proper
	symlink support. On systems that support symlinks, we can generate a
	file into a staging area then symlink it into the final directory very
	quickly. On Windows, we have to perform a full file copy. This incurs
	much more I/O. And if done poorly, can muck with file modification
	times, messing up build dependencies. As of the summer of 2013, the
	impact of symlinks is being mitigated through the use
	of an :term:`install manifest`.

	These issues impact both clobber and incremental builds.

	Recursive make traversal is slow
	================================

	The build system has traditionally been built by employing recursive
	make. Recursive make involves make iterating through directories / make
	files sequentially and executing each in turn. This is inefficient for
	directories containing few targets/tasks because make could be starved
	for work when processing these directories. Any time make is starved,
	the build isn't using all available CPU cycles and the build is slower
	as a result.

	Work has started in bug 907365 to fix this issue by changing the way
	make traverses all the make files.

	The impact of slow recursive make traversal is mostly felt on
	incremental builds. Traditionally, most of the wall time during a
	no-op build is spent in make traversal.

	make is inefficient
	===================

	Compared to modern build backends like Tup or Ninja, make is slow and
	inefficient. We can only make make so fast. At some point, we'll hit a
	performance plateau and will need to use a different tool to make builds
	faster.

	Please note that clobber and incremental builds are different. A clobber
	build with make will likely be as fast as a clobber build with e.g. Tup.
	However, Tup should vastly outperform make when it comes to incremental
	builds. Therefore, this issue is mostly seen when performing incremental
	builds.

	C++ header dependency hell
	==========================

	Modifying a .h file can have significant impact on the build system.
	If you modify a .h that is used by 1000 C++ files, all of those 1000
	C++ files will be recompiled.

	Our code base has traditionally been sloppy managing the impact of
	changed headers on build performance. Bug 785103 tracks improving the
	situation.

	This issue mostly impacts the times of an :term:`incremental build`.

	A search/indexing service on your machine is running
	====================================================

	Many operating systems have a background service that automatically
	indexes filesystem content to make searching faster. On Windows, you
	have the Windows Search Service. On OS X, you have Finder.

	These background services sometimes take a keen interest in the files
	being produced as part of the build. Since the build system produces
	hundreds of megabytes or even a few gigabytes of file data, you can
	imagine how much work this is to index! If this work is being performed
	while the build is running, your build will be slower.

	OS X's Finder is notorious for indexing when the build is running. And,
	it has a tendency to suck up a whole CPU core. This can make builds
	several minutes slower. If you build with ``mach`` and have the optional
	``psutil`` package built (it requires Python development headers - see
	:ref:`python` for more) and Finder is running during a build, mach will
	print a warning at the end of the build, complete with instructions on
	how to fix it.