tree b55ad7e85222381c37bc143d7f45187e9a469be6
parent 992e927e217baa8a74e6e2c5d7417cb65cf24824
author David 'Digit' Turner <digit@google.com> 1580749932 +0100
committer Commit Bot <commit-bot@chromium.org> 1583947540 +0000

Reduce RAM usage and improve speed of "gn gen"

This CL continues previous optimization work [1] to improve
GN's speed and RAM usage, especially for the Zircon build
(see [2] for measurements).

It consists in the following changes:

- Introduce src/gn/hash_table_base.h to generalize
  the fast hash table used by StringAtom's implementation
  through a custom template class, and document its
  usage properly (with a disclaimer not to use it
  unless profiling shows it's worthwhile).

- Use it to simplify and optimize the implementation
  of the UniqueVector<> template.

- Implement SourceDir and Label with StringAtom instead
  of std::string instances.

+ Remove a small unwanted const Label copy in
  src/gn/analyzer.cc

[1] https://gn-review.googlesource.com/c/gn/+/7280

[2] Measurement results for "gn gen" runs on my
    local machine, both using recent Chromium and Fuchsia
    source checkouts.

 CHROMIUM: RAM saved:  68 MiB  Speedup:  8.4%
 ZIRCON:   RAM saved: 405 MiB  Speedup: 24.5%

// CHROMIUM BEFORE

/work/chromium0/src$ repeat_cmd 5  /tmp/gn-master gen out/default/
Done. Made 12150 targets from 2121 files in 4256ms
Done. Made 12150 targets from 2121 files in 4218ms
Done. Made 12150 targets from 2121 files in 4286ms
Done. Made 12150 targets from 2121 files in 4201ms *
Done. Made 12150 targets from 2121 files in 4244ms

/work/chromium0/src$ /usr/bin/time -v /tmp/gn-master gen out/default
Maximum resident set size (kbytes): 535480

// CHROMIUM AFTER

/work/chromium0/src$ repeat_cmd 5  /tmp/gn-after gen out/default/
Done. Made 12150 targets from 2121 files in 3992ms
Done. Made 12150 targets from 2121 files in 3874ms *
Done. Made 12150 targets from 2121 files in 3951ms
Done. Made 12150 targets from 2121 files in 3907ms
Done. Made 12150 targets from 2121 files in 3905ms

/work/chromium0/src$ /usr/bin/time -v /tmp/gn-after gen out/default
Maximum resident set size (kbytes): 466056

// ZIRCON BEFORE

/work/fuchsia0$ repeat_cmd 5 /tmp/gn-master gen --root=zircon out/default.zircon
Done. Made 47910 targets from 782 files in 8915ms *
Done. Made 47910 targets from 782 files in 8945ms
Done. Made 47910 targets from 782 files in 9141ms
Done. Made 47910 targets from 782 files in 9094ms
Done. Made 47910 targets from 782 files in 9512ms

/work/fuchsia0$ /usr/bin/time -v  /tmp/gn-master gen --root=zircon out/default.zircon
Maximum resident set size (kbytes): 1595364

// ZIRCON AFTER

/work/fuchsia0$ repeat_cmd 5 /tmp/gn-after gen --root=zircon out/default.zircon
Done. Made 47910 targets from 782 files in 7156ms *
Done. Made 47910 targets from 782 files in 7715ms
Done. Made 47910 targets from 782 files in 7327ms
Done. Made 47910 targets from 782 files in 7491ms
Done. Made 47910 targets from 782 files in 7573ms

/work/fuchsia0$ /usr/bin/time -v  /tmp/gn-master gen --root=zircon out/default.zircon
Maximum resident set size (kbytes): 1180640

Change-Id: Id315564a9b2285cd89ca07cd69962f008439c9fb
Reviewed-on: https://gn-review.googlesource.com/c/gn/+/7340
Reviewed-by: Brett Wilson <brettw@chromium.org>
Reviewed-by: Brett Wilson <brettw@google.com>
Commit-Queue: Brett Wilson <brettw@chromium.org>