tree 2f02d4b0ed7da9601c9f8753480c309b391b3b01
parent 7c5d1002f4de08ca336a1bfee12272f875a60a66
author David 'Digit' Turner <digit@google.com> 1585912606 +0200
committer Commit Bot <commit-bot@chromium.org> 1588682339 +0000

Speedup --json=ide and --export-compile-commands=default

This CL significantly speeds up the generation of the project.json,
as well as that of compiled_commands.json. Note that these files
can be very large (e.g. 74 MiB / 6.4 MiB for Chromium, and
139 MiB / 45 MiB for Fuchsia), and are required by every Fuchsia
'fx gen' invocation.

Measurements [1] shows 2 seconds saved in the Fuchsia case, and
0.5s in the Chromium one. More importantly, this reduces peak
RSS usage by 323 MiB on Chromium, and 515 MiB on Fuchsia.

- compile_commands_writer.cc: output content directly to an
  std::ostream instead of using an intermediate std::string,
  using a StringBufferOutput for storage / file comparison /
  file write.

- json_project_writer.cc: also use an std::ostream instead of
  using an intermediate std::string, as well as implement its
  own simple JSON writer class that doesn't require building
  a complete in-memory representation of the whole document
  before converting it into a string.

- rust_project_writer.cc: use StringOutputBuffer to avoid
  touching the file if the content didn't change. Also
  provide a minimal speedup (e.g. 90ms -> 70ms).

[1] Measurements are performed against the ToT GN (gn-master)
    and the optimized one that results from this CL and 2
    preparatory parent CLs (gn-json-writer).

    Both are built with --use-icf --use-lto with the same
    recent Clang toolchain. Tests are run on a Linux
    workstation.

//
// CHROMIUM BEFORE
//
// NOTE: Command outputs were-reordered for better readability.
//
/work/chromium0/src$ repeat_cmd 5 /tmp/gn-master gen --ide=json
--export-compile-commands=default out/Release
Generating JSON projects took 1989ms (best)
Generating JSON projects took 2146ms
Generating JSON projects took 1993ms
Generating JSON projects took 2017ms
Generating JSON projects took 2167ms
Generating compile_commands took 45ms
Generating compile_commands took 61ms
Generating compile_commands took 41ms (best)
Generating compile_commands took 70ms
Generating compile_commands took 71ms
Done. Made 13413 targets from 2220 files in 7476ms
Done. Made 13413 targets from 2220 files in 7140ms
Done. Made 13413 targets from 2220 files in 6780ms (best)
Done. Made 13413 targets from 2220 files in 7022ms
Done. Made 13413 targets from 2220 files in 7203ms

//
// CHROMIUM AFTER
//
//  0.5s faster, 323 MiB less RAM.
//
/work/chromium0/src$ repeat_cmd 5 /tmp/gn-json-writer gen --ide=json
--export-compile-commands=default out/Release
Generating JSON projects took 1442ms
Generating JSON projects took 1523ms
Generating JSON projects took 1422ms (best)
Generating JSON projects took 1591ms
Generating JSON projects took 1579ms
Generating compile_commands took 24ms
Generating compile_commands took 25ms
Generating compile_commands took 23ms (best)
Generating compile_commands took 35ms
Generating compile_commands took 45ms
Done. Made 13413 targets from 2220 files in 6284ms
Done. Made 13413 targets from 2220 files in 6425ms
Done. Made 13413 targets from 2220 files in 6265ms (best)
Done. Made 13413 targets from 2220 files in 6434ms
Done. Made 13413 targets from 2220 files in 6541ms

/work/chromium0/src$ /usr/bin/time -f %M /tmp/gn-master gen --ide=json
--export-compile-commands=default out/Release
...
910784

/work/chromium0/src$ /usr/bin/time -f %M /tmp/gn-json-writer gen
--ide=json --export-compile-commands=default out/Release
...
579244

//
// FUCHSIA BEFORE
//

/work/fx-gn$ cp -f /tmp/gn-master prebuilt/third_party/gn/linux-x64/gn
/work/fx-gn$ repeat_cmd 5 prebuilt/third_party/gn/linux-x64/gn gen --ide=json --export-compile-commands=default out/default
Generating JSON projects took 4979ms
Generating JSON projects took 4860ms
Generating JSON projects took 5603ms
Generating JSON projects took 4683ms (best)
Generating JSON projects took 5346ms
Generating compile_commands took 655ms
Generating compile_commands took 589ms (best)
Generating compile_commands took 701ms
Generating compile_commands took 601ms
Generating compile_commands took 668ms
Done. Made 41018 targets from 3754 files in 18888ms
Done. Made 41018 targets from 3754 files in 18969ms
Done. Made 41018 targets from 3754 files in 20393ms
Done. Made 41018 targets from 3754 files in 18316ms (best)
Done. Made 41018 targets from 3754 files in 19534ms

//
// FUCHSIA AFTER
//
// 2s faster, 505 MiB less RAM.
//

/work/fx-gn$ cp -f /tmp/gn-json-writer prebuilt/third_party/gn/linux-x64/gn
/work/fx-gn$ repeat_cmd 5 prebuilt/third_party/gn/linux-x64/gn gen --ide=json --export-compile-commands=default out/default
Generating JSON projects took 2835ms (best)
Generating JSON projects took 2893ms
Generating JSON projects took 2877ms
Generating JSON projects took 3211ms
Generating JSON projects took 3073ms
Generating compile_commands took 399ms (best)
Generating compile_commands took 426ms
Generating compile_commands took 415ms
Generating compile_commands took 443ms
Generating compile_commands took 438ms
Done. Made 41018 targets from 3754 files in 16504ms
Done. Made 41018 targets from 3754 files in 16323ms (best)
Done. Made 41018 targets from 3754 files in 16669ms
Done. Made 41018 targets from 3754 files in 17366ms
Done. Made 41018 targets from 3754 files in 16698ms

/work/fx-gn$ /usr/bin/time -f %M prebuilt/third_party/gn/linux-x64/gn gen --ide=json --export-compile-commands=default out/default
...
1546356

/work/fx-gn$ /usr/bin/time -f %M prebuilt/third_party/gn/linux-x64/gn gen --ide=json --export-compile-commands=default out/default
...
1028656

Change-Id: Ie1a4dcd11f0798ca70b42f67f76a37bd2ea5c94a
Reviewed-on: https://gn-review.googlesource.com/c/gn/+/7942
Commit-Queue: David Turner <digit@google.com>
Reviewed-by: Scott Graham <scottmg@chromium.org>