)]}' { "commit": "6ed85ec5155d9933b44a4aa5e132c9833090a6ca", "tree": "2f02d4b0ed7da9601c9f8753480c309b391b3b01", "parents": [ "7c5d1002f4de08ca336a1bfee12272f875a60a66" ], "author": { "name": "David \u0027Digit\u0027 Turner", "email": "digit@google.com", "time": "Fri Apr 03 13:16:46 2020 +0200" }, "committer": { "name": "Commit Bot", "email": "commit-bot@chromium.org", "time": "Tue May 05 12:38:59 2020 +0000" }, "message": "Speedup --json\u003dide and --export-compile-commands\u003ddefault\n\nThis CL significantly speeds up the generation of the project.json,\nas well as that of compiled_commands.json. Note that these files\ncan be very large (e.g. 74 MiB / 6.4 MiB for Chromium, and\n139 MiB / 45 MiB for Fuchsia), and are required by every Fuchsia\n\u0027fx gen\u0027 invocation.\n\nMeasurements [1] shows 2 seconds saved in the Fuchsia case, and\n0.5s in the Chromium one. More importantly, this reduces peak\nRSS usage by 323 MiB on Chromium, and 515 MiB on Fuchsia.\n\n- compile_commands_writer.cc: output content directly to an\n std::ostream instead of using an intermediate std::string,\n using a StringBufferOutput for storage / file comparison /\n file write.\n\n- json_project_writer.cc: also use an std::ostream instead of\n using an intermediate std::string, as well as implement its\n own simple JSON writer class that doesn\u0027t require building\n a complete in-memory representation of the whole document\n before converting it into a string.\n\n- rust_project_writer.cc: use StringOutputBuffer to avoid\n touching the file if the content didn\u0027t change. Also\n provide a minimal speedup (e.g. 90ms -\u003e 70ms).\n\n[1] Measurements are performed against the ToT GN (gn-master)\n and the optimized one that results from this CL and 2\n preparatory parent CLs (gn-json-writer).\n\n Both are built with --use-icf --use-lto with the same\n recent Clang toolchain. Tests are run on a Linux\n workstation.\n\n//\n// CHROMIUM BEFORE\n//\n// NOTE: Command outputs were-reordered for better readability.\n//\n/work/chromium0/src$ repeat_cmd 5 /tmp/gn-master gen --ide\u003djson\n--export-compile-commands\u003ddefault out/Release\nGenerating JSON projects took 1989ms (best)\nGenerating JSON projects took 2146ms\nGenerating JSON projects took 1993ms\nGenerating JSON projects took 2017ms\nGenerating JSON projects took 2167ms\nGenerating compile_commands took 45ms\nGenerating compile_commands took 61ms\nGenerating compile_commands took 41ms (best)\nGenerating compile_commands took 70ms\nGenerating compile_commands took 71ms\nDone. Made 13413 targets from 2220 files in 7476ms\nDone. Made 13413 targets from 2220 files in 7140ms\nDone. Made 13413 targets from 2220 files in 6780ms (best)\nDone. Made 13413 targets from 2220 files in 7022ms\nDone. Made 13413 targets from 2220 files in 7203ms\n\n//\n// CHROMIUM AFTER\n//\n// 0.5s faster, 323 MiB less RAM.\n//\n/work/chromium0/src$ repeat_cmd 5 /tmp/gn-json-writer gen --ide\u003djson\n--export-compile-commands\u003ddefault out/Release\nGenerating JSON projects took 1442ms\nGenerating JSON projects took 1523ms\nGenerating JSON projects took 1422ms (best)\nGenerating JSON projects took 1591ms\nGenerating JSON projects took 1579ms\nGenerating compile_commands took 24ms\nGenerating compile_commands took 25ms\nGenerating compile_commands took 23ms (best)\nGenerating compile_commands took 35ms\nGenerating compile_commands took 45ms\nDone. Made 13413 targets from 2220 files in 6284ms\nDone. Made 13413 targets from 2220 files in 6425ms\nDone. Made 13413 targets from 2220 files in 6265ms (best)\nDone. Made 13413 targets from 2220 files in 6434ms\nDone. Made 13413 targets from 2220 files in 6541ms\n\n/work/chromium0/src$ /usr/bin/time -f %M /tmp/gn-master gen --ide\u003djson\n--export-compile-commands\u003ddefault out/Release\n...\n910784\n\n/work/chromium0/src$ /usr/bin/time -f %M /tmp/gn-json-writer gen\n--ide\u003djson --export-compile-commands\u003ddefault out/Release\n...\n579244\n\n//\n// FUCHSIA BEFORE\n//\n\n/work/fx-gn$ cp -f /tmp/gn-master prebuilt/third_party/gn/linux-x64/gn\n/work/fx-gn$ repeat_cmd 5 prebuilt/third_party/gn/linux-x64/gn gen --ide\u003djson --export-compile-commands\u003ddefault out/default\nGenerating JSON projects took 4979ms\nGenerating JSON projects took 4860ms\nGenerating JSON projects took 5603ms\nGenerating JSON projects took 4683ms (best)\nGenerating JSON projects took 5346ms\nGenerating compile_commands took 655ms\nGenerating compile_commands took 589ms (best)\nGenerating compile_commands took 701ms\nGenerating compile_commands took 601ms\nGenerating compile_commands took 668ms\nDone. Made 41018 targets from 3754 files in 18888ms\nDone. Made 41018 targets from 3754 files in 18969ms\nDone. Made 41018 targets from 3754 files in 20393ms\nDone. Made 41018 targets from 3754 files in 18316ms (best)\nDone. Made 41018 targets from 3754 files in 19534ms\n\n//\n// FUCHSIA AFTER\n//\n// 2s faster, 505 MiB less RAM.\n//\n\n/work/fx-gn$ cp -f /tmp/gn-json-writer prebuilt/third_party/gn/linux-x64/gn\n/work/fx-gn$ repeat_cmd 5 prebuilt/third_party/gn/linux-x64/gn gen --ide\u003djson --export-compile-commands\u003ddefault out/default\nGenerating JSON projects took 2835ms (best)\nGenerating JSON projects took 2893ms\nGenerating JSON projects took 2877ms\nGenerating JSON projects took 3211ms\nGenerating JSON projects took 3073ms\nGenerating compile_commands took 399ms (best)\nGenerating compile_commands took 426ms\nGenerating compile_commands took 415ms\nGenerating compile_commands took 443ms\nGenerating compile_commands took 438ms\nDone. Made 41018 targets from 3754 files in 16504ms\nDone. Made 41018 targets from 3754 files in 16323ms (best)\nDone. Made 41018 targets from 3754 files in 16669ms\nDone. Made 41018 targets from 3754 files in 17366ms\nDone. Made 41018 targets from 3754 files in 16698ms\n\n/work/fx-gn$ /usr/bin/time -f %M prebuilt/third_party/gn/linux-x64/gn gen --ide\u003djson --export-compile-commands\u003ddefault out/default\n...\n1546356\n\n/work/fx-gn$ /usr/bin/time -f %M prebuilt/third_party/gn/linux-x64/gn gen --ide\u003djson --export-compile-commands\u003ddefault out/default\n...\n1028656\n\nChange-Id: Ie1a4dcd11f0798ca70b42f67f76a37bd2ea5c94a\nReviewed-on: https://gn-review.googlesource.com/c/gn/+/7942\nCommit-Queue: David Turner \u003cdigit@google.com\u003e\nReviewed-by: Scott Graham \u003cscottmg@chromium.org\u003e\n", "tree_diff": [ { "type": "modify", "old_id": "5dfd2f69fc1a427e8cf670dc6e05db96026de2ba", "old_mode": 33188, "old_path": "src/gn/compile_commands_writer.cc", "new_id": "df5e73f3b78219588531ca5a212a0e282b12b8fc", "new_mode": 33188, "new_path": "src/gn/compile_commands_writer.cc" }, { "type": "modify", "old_id": "78db51e59cbc670db58d4884522faebfdf1a2641", "old_mode": 33188, "old_path": "src/gn/compile_commands_writer.h", "new_id": "be4e90bc9a0c1471f1ace15389b0bab481a7783c", "new_mode": 33188, "new_path": "src/gn/compile_commands_writer.h" }, { "type": "modify", "old_id": "99dc55d62478cb12282aca5f448d642437ef1865", "old_mode": 33188, "old_path": "src/gn/compile_commands_writer_unittest.cc", "new_id": "9bbb076ac6b2d3cb64042479e9a0c4d7abb29daf", "new_mode": 33188, "new_path": "src/gn/compile_commands_writer_unittest.cc" }, { "type": "modify", "old_id": "c478379cbe98bccf1b1f1f2595e9d2cc235f8ccd", "old_mode": 33188, "old_path": "src/gn/json_project_writer.cc", "new_id": "0884aa73cd06159f3a323d68b4b91334a1921cf4", "new_mode": 33188, "new_path": "src/gn/json_project_writer.cc" }, { "type": "modify", "old_id": "9d396d327f48edac26164ec5b3a16b2a78ba7f4e", "old_mode": 33188, "old_path": "src/gn/json_project_writer.h", "new_id": "74293a494a52e4d5ca96cb09be0fe44cc740bbd0", "new_mode": 33188, "new_path": "src/gn/json_project_writer.h" }, { "type": "modify", "old_id": "754f9b6a7c073c0e352562cf53bd9015695c261c", "old_mode": 33188, "old_path": "src/gn/rust_project_writer.cc", "new_id": "19eb42605ca29c22ea04b3d935da19d431d31e2e", "new_mode": 33188, "new_path": "src/gn/rust_project_writer.cc" } ] }