Speedup --json=ide and --export-compile-commands=default

This CL significantly speeds up the generation of the project.json,
as well as that of compiled_commands.json. Note that these files
can be very large (e.g. 74 MiB / 6.4 MiB for Chromium, and
139 MiB / 45 MiB for Fuchsia), and are required by every Fuchsia
'fx gen' invocation.

Measurements [1] shows 2 seconds saved in the Fuchsia case, and
0.5s in the Chromium one. More importantly, this reduces peak
RSS usage by 323 MiB on Chromium, and 515 MiB on Fuchsia.

- compile_commands_writer.cc: output content directly to an
  std::ostream instead of using an intermediate std::string,
  using a StringBufferOutput for storage / file comparison /
  file write.

- json_project_writer.cc: also use an std::ostream instead of
  using an intermediate std::string, as well as implement its
  own simple JSON writer class that doesn't require building
  a complete in-memory representation of the whole document
  before converting it into a string.

- rust_project_writer.cc: use StringOutputBuffer to avoid
  touching the file if the content didn't change. Also
  provide a minimal speedup (e.g. 90ms -> 70ms).

[1] Measurements are performed against the ToT GN (gn-master)
    and the optimized one that results from this CL and 2
    preparatory parent CLs (gn-json-writer).

    Both are built with --use-icf --use-lto with the same
    recent Clang toolchain. Tests are run on a Linux
    workstation.

//
// CHROMIUM BEFORE
//
// NOTE: Command outputs were-reordered for better readability.
//
/work/chromium0/src$ repeat_cmd 5 /tmp/gn-master gen --ide=json
--export-compile-commands=default out/Release
Generating JSON projects took 1989ms (best)
Generating JSON projects took 2146ms
Generating JSON projects took 1993ms
Generating JSON projects took 2017ms
Generating JSON projects took 2167ms
Generating compile_commands took 45ms
Generating compile_commands took 61ms
Generating compile_commands took 41ms (best)
Generating compile_commands took 70ms
Generating compile_commands took 71ms
Done. Made 13413 targets from 2220 files in 7476ms
Done. Made 13413 targets from 2220 files in 7140ms
Done. Made 13413 targets from 2220 files in 6780ms (best)
Done. Made 13413 targets from 2220 files in 7022ms
Done. Made 13413 targets from 2220 files in 7203ms

//
// CHROMIUM AFTER
//
//  0.5s faster, 323 MiB less RAM.
//
/work/chromium0/src$ repeat_cmd 5 /tmp/gn-json-writer gen --ide=json
--export-compile-commands=default out/Release
Generating JSON projects took 1442ms
Generating JSON projects took 1523ms
Generating JSON projects took 1422ms (best)
Generating JSON projects took 1591ms
Generating JSON projects took 1579ms
Generating compile_commands took 24ms
Generating compile_commands took 25ms
Generating compile_commands took 23ms (best)
Generating compile_commands took 35ms
Generating compile_commands took 45ms
Done. Made 13413 targets from 2220 files in 6284ms
Done. Made 13413 targets from 2220 files in 6425ms
Done. Made 13413 targets from 2220 files in 6265ms (best)
Done. Made 13413 targets from 2220 files in 6434ms
Done. Made 13413 targets from 2220 files in 6541ms

/work/chromium0/src$ /usr/bin/time -f %M /tmp/gn-master gen --ide=json
--export-compile-commands=default out/Release
...
910784

/work/chromium0/src$ /usr/bin/time -f %M /tmp/gn-json-writer gen
--ide=json --export-compile-commands=default out/Release
...
579244

//
// FUCHSIA BEFORE
//

/work/fx-gn$ cp -f /tmp/gn-master prebuilt/third_party/gn/linux-x64/gn
/work/fx-gn$ repeat_cmd 5 prebuilt/third_party/gn/linux-x64/gn gen --ide=json --export-compile-commands=default out/default
Generating JSON projects took 4979ms
Generating JSON projects took 4860ms
Generating JSON projects took 5603ms
Generating JSON projects took 4683ms (best)
Generating JSON projects took 5346ms
Generating compile_commands took 655ms
Generating compile_commands took 589ms (best)
Generating compile_commands took 701ms
Generating compile_commands took 601ms
Generating compile_commands took 668ms
Done. Made 41018 targets from 3754 files in 18888ms
Done. Made 41018 targets from 3754 files in 18969ms
Done. Made 41018 targets from 3754 files in 20393ms
Done. Made 41018 targets from 3754 files in 18316ms (best)
Done. Made 41018 targets from 3754 files in 19534ms

//
// FUCHSIA AFTER
//
// 2s faster, 505 MiB less RAM.
//

/work/fx-gn$ cp -f /tmp/gn-json-writer prebuilt/third_party/gn/linux-x64/gn
/work/fx-gn$ repeat_cmd 5 prebuilt/third_party/gn/linux-x64/gn gen --ide=json --export-compile-commands=default out/default
Generating JSON projects took 2835ms (best)
Generating JSON projects took 2893ms
Generating JSON projects took 2877ms
Generating JSON projects took 3211ms
Generating JSON projects took 3073ms
Generating compile_commands took 399ms (best)
Generating compile_commands took 426ms
Generating compile_commands took 415ms
Generating compile_commands took 443ms
Generating compile_commands took 438ms
Done. Made 41018 targets from 3754 files in 16504ms
Done. Made 41018 targets from 3754 files in 16323ms (best)
Done. Made 41018 targets from 3754 files in 16669ms
Done. Made 41018 targets from 3754 files in 17366ms
Done. Made 41018 targets from 3754 files in 16698ms

/work/fx-gn$ /usr/bin/time -f %M prebuilt/third_party/gn/linux-x64/gn gen --ide=json --export-compile-commands=default out/default
...
1546356

/work/fx-gn$ /usr/bin/time -f %M prebuilt/third_party/gn/linux-x64/gn gen --ide=json --export-compile-commands=default out/default
...
1028656

Change-Id: Ie1a4dcd11f0798ca70b42f67f76a37bd2ea5c94a
Reviewed-on: https://gn-review.googlesource.com/c/gn/+/7942
Commit-Queue: David Turner <digit@google.com>
Reviewed-by: Scott Graham <scottmg@chromium.org>
diff --git a/src/gn/compile_commands_writer.cc b/src/gn/compile_commands_writer.cc
index 5dfd2f6..df5e73f 100644
--- a/src/gn/compile_commands_writer.cc
+++ b/src/gn/compile_commands_writer.cc
@@ -18,6 +18,7 @@
 #include "gn/filesystem_utils.h"
 #include "gn/ninja_target_command_util.h"
 #include "gn/path_output.h"
+#include "gn/string_output_buffer.h"
 #include "gn/substitution_writer.h"
 
 // Structure of JSON output file
@@ -51,6 +52,22 @@
   std::string frameworks;
 };
 
+// Helper template function to call RecursiveTargetConfigToStream<std::string>
+// and return the JSON-escaped resulting string.
+//
+// NOTE: The Windows compiler cannot properly deduce the first parameter type
+// so pass it at each call site to ensure proper builds for this platform.
+template <typename T, typename Writer>
+std::string FlagsGetter(const Target* target,
+                        const std::vector<T>& (ConfigValues::*getter)() const,
+                        const Writer& writer) {
+  std::string result;
+  std::ostringstream out;
+  RecursiveTargetConfigToStream<T>(target, getter, writer, out);
+  base::EscapeJSONString(out.str(), false, &result);
+  return result;
+};
+
 void SetupCompileFlags(const Target* target,
                        PathOutput& path_output,
                        EscapeOptions opts,
@@ -58,78 +75,66 @@
   bool has_precompiled_headers =
       target->config_values().has_precompiled_headers();
 
-  std::ostringstream defines_out;
-  RecursiveTargetConfigToStream<std::string>(target, &ConfigValues::defines,
-                                             DefineWriter(ESCAPE_SPACE, true),
-                                             defines_out);
-  base::EscapeJSONString(defines_out.str(), false, &flags.defines);
+  flags.defines = FlagsGetter<std::string>(target, &ConfigValues::defines,
+                                           DefineWriter(ESCAPE_SPACE, true));
 
-  std::ostringstream framework_dirs_out;
-  RecursiveTargetConfigToStream<SourceDir>(
-      target, &ConfigValues::framework_dirs,
-      FrameworkDirsWriter(path_output, "-F"), framework_dirs_out);
-  base::EscapeJSONString(framework_dirs_out.str(), false,
-                         &flags.framework_dirs);
+  flags.framework_dirs =
+      FlagsGetter<SourceDir>(target, &ConfigValues::framework_dirs,
+                             FrameworkDirsWriter(path_output, "-F"));
 
-  std::ostringstream frameworks_out;
-  RecursiveTargetConfigToStream<std::string>(
+  flags.frameworks = FlagsGetter<std::string>(
       target, &ConfigValues::frameworks,
-      FrameworksWriter(ESCAPE_SPACE, true, "-framework "), frameworks_out);
-  base::EscapeJSONString(frameworks_out.str(), false, &flags.frameworks);
+      FrameworksWriter(ESCAPE_SPACE, true, "-framework"));
 
-  std::ostringstream includes_out;
-  RecursiveTargetConfigToStream<SourceDir>(target, &ConfigValues::include_dirs,
-                                           IncludeWriter(path_output),
-                                           includes_out);
-  base::EscapeJSONString(includes_out.str(), false, &flags.includes);
+  flags.includes = FlagsGetter<SourceDir>(target, &ConfigValues::include_dirs,
+                                          IncludeWriter(path_output));
 
-  std::ostringstream cflags_out;
-  WriteOneFlag(target, &CSubstitutionCFlags, false, Tool::kToolNone,
-               &ConfigValues::cflags, opts, path_output, cflags_out,
-               /*write_substitution=*/false);
-  base::EscapeJSONString(cflags_out.str(), false, &flags.cflags);
+  // Helper lambda to call WriteOneFlag() and return the resulting
+  // escaped JSON string.
+  auto one_flag = [&](const Substitution* substitution,
+                      bool has_precompiled_headers, const char* tool_name,
+                      const std::vector<std::string>& (ConfigValues::*getter)()
+                          const) -> std::string {
+    std::string result;
+    std::ostringstream out;
+    WriteOneFlag(target, substitution, has_precompiled_headers, tool_name,
+                 getter, opts, path_output, out, /*write_substitution=*/false);
+    base::EscapeJSONString(out.str(), false, &result);
+    return result;
+  };
 
-  std::ostringstream cflags_c_out;
-  WriteOneFlag(target, &CSubstitutionCFlagsC, has_precompiled_headers,
-               CTool::kCToolCc, &ConfigValues::cflags_c, opts, path_output,
-               cflags_c_out, /*write_substitution=*/false);
-  base::EscapeJSONString(cflags_c_out.str(), false, &flags.cflags_c);
+  flags.cflags = one_flag(&CSubstitutionCFlags, false, Tool::kToolNone,
+                          &ConfigValues::cflags);
 
-  std::ostringstream cflags_cc_out;
-  WriteOneFlag(target, &CSubstitutionCFlagsCc, has_precompiled_headers,
-               CTool::kCToolCxx, &ConfigValues::cflags_cc, opts, path_output,
-               cflags_cc_out, /*write_substitution=*/false);
-  base::EscapeJSONString(cflags_cc_out.str(), false, &flags.cflags_cc);
+  flags.cflags_c = one_flag(&CSubstitutionCFlagsC, has_precompiled_headers,
+                            CTool::kCToolCc, &ConfigValues::cflags_c);
 
-  std::ostringstream cflags_objc_out;
-  WriteOneFlag(target, &CSubstitutionCFlagsObjC, has_precompiled_headers,
-               CTool::kCToolObjC, &ConfigValues::cflags_objc, opts, path_output,
-               cflags_objc_out,
-               /*write_substitution=*/false);
-  base::EscapeJSONString(cflags_objc_out.str(), false, &flags.cflags_objc);
+  flags.cflags_cc = one_flag(&CSubstitutionCFlagsCc, has_precompiled_headers,
+                             CTool::kCToolCxx, &ConfigValues::cflags_cc);
 
-  std::ostringstream cflags_objcc_out;
-  WriteOneFlag(target, &CSubstitutionCFlagsObjCc, has_precompiled_headers,
-               CTool::kCToolObjCxx, &ConfigValues::cflags_objcc, opts,
-               path_output, cflags_objcc_out, /*write_substitution=*/false);
-  base::EscapeJSONString(cflags_objcc_out.str(), false, &flags.cflags_objcc);
+  flags.cflags_objc =
+      one_flag(&CSubstitutionCFlagsObjC, has_precompiled_headers,
+               CTool::kCToolObjC, &ConfigValues::cflags_objc);
+
+  flags.cflags_objcc =
+      one_flag(&CSubstitutionCFlagsObjCc, has_precompiled_headers,
+               CTool::kCToolObjCxx, &ConfigValues::cflags_objcc);
 }
 
 void WriteFile(const SourceFile& source,
                PathOutput& path_output,
-               std::string* compile_commands) {
+               std::ostream& out) {
   std::ostringstream rel_source_path;
-  path_output.WriteFile(rel_source_path, source);
-  compile_commands->append("    \"file\": \"");
-  compile_commands->append(rel_source_path.str());
+  out << "    \"file\": \"";
+  path_output.WriteFile(out, source);
 }
 
-void WriteDirectory(std::string build_dir, std::string* compile_commands) {
-  compile_commands->append("\",");
-  compile_commands->append(kPrettyPrintLineEnding);
-  compile_commands->append("    \"directory\": \"");
-  compile_commands->append(build_dir);
-  compile_commands->append("\",");
+void WriteDirectory(std::string build_dir, std::ostream& out) {
+  out << "\",";
+  out << kPrettyPrintLineEnding;
+  out << "    \"directory\": \"";
+  out << build_dir;
+  out << "\",";
 }
 
 void WriteCommand(const Target* target,
@@ -140,40 +145,42 @@
                   SourceFile::Type source_type,
                   const char* tool_name,
                   EscapeOptions opts,
-                  std::string* compile_commands) {
+                  std::ostream& out) {
   EscapeOptions no_quoting(opts);
   no_quoting.inhibit_quoting = true;
   const Tool* tool = target->toolchain()->GetTool(tool_name);
-  std::ostringstream command_out;
+
+  out << kPrettyPrintLineEnding;
+  out << "    \"command\": \"";
 
   for (const auto& range : tool->command().ranges()) {
     // TODO: this is emitting a bonus space prior to each substitution.
     if (range.type == &SubstitutionLiteral) {
-      EscapeStringToStream(command_out, range.literal, no_quoting);
+      EscapeStringToStream(out, range.literal, no_quoting);
     } else if (range.type == &SubstitutionOutput) {
-      path_output.WriteFiles(command_out, tool_outputs);
+      path_output.WriteFiles(out, tool_outputs);
     } else if (range.type == &CSubstitutionDefines) {
-      command_out << flags.defines;
+      out << flags.defines;
     } else if (range.type == &CSubstitutionFrameworkDirs) {
-      command_out << flags.framework_dirs;
+      out << flags.framework_dirs;
     } else if (range.type == &CSubstitutionFrameworks) {
-      command_out << flags.frameworks;
+      out << flags.frameworks;
     } else if (range.type == &CSubstitutionIncludeDirs) {
-      command_out << flags.includes;
+      out << flags.includes;
     } else if (range.type == &CSubstitutionCFlags) {
-      command_out << flags.cflags;
+      out << flags.cflags;
     } else if (range.type == &CSubstitutionCFlagsC) {
       if (source_type == SourceFile::SOURCE_C)
-        command_out << flags.cflags_c;
+        out << flags.cflags_c;
     } else if (range.type == &CSubstitutionCFlagsCc) {
       if (source_type == SourceFile::SOURCE_CPP)
-        command_out << flags.cflags_cc;
+        out << flags.cflags_cc;
     } else if (range.type == &CSubstitutionCFlagsObjC) {
       if (source_type == SourceFile::SOURCE_M)
-        command_out << flags.cflags_objc;
+        out << flags.cflags_objc;
     } else if (range.type == &CSubstitutionCFlagsObjCc) {
       if (source_type == SourceFile::SOURCE_MM)
-        command_out << flags.cflags_objcc;
+        out << flags.cflags_objcc;
     } else if (range.type == &SubstitutionLabel ||
                range.type == &SubstitutionLabelName ||
                range.type == &SubstitutionRootGenDir ||
@@ -189,7 +196,7 @@
                range.type == &SubstitutionSourceGenDir ||
                range.type == &SubstitutionSourceOutDir ||
                range.type == &SubstitutionSourceTargetRelative) {
-      EscapeStringToStream(command_out,
+      EscapeStringToStream(out,
                            SubstitutionWriter::GetCompilerSubstitution(
                                target, source, range.type),
                            opts);
@@ -201,20 +208,13 @@
       continue;
     }
   }
-  compile_commands->append(kPrettyPrintLineEnding);
-  compile_commands->append("    \"command\": \"");
-  compile_commands->append(command_out.str());
 }
 
-}  // namespace
-
-void CompileCommandsWriter::RenderJSON(const BuildSettings* build_settings,
-                                       std::vector<const Target*>& all_targets,
-                                       std::string* compile_commands) {
-  // TODO: Determine out an appropriate size to reserve.
-  compile_commands->reserve(all_targets.size() * 100);
-  compile_commands->append("[");
-  compile_commands->append(kPrettyPrintLineEnding);
+void OutputJSON(const BuildSettings* build_settings,
+                std::vector<const Target*>& all_targets,
+                std::ostream& out) {
+  out << '[';
+  out << kPrettyPrintLineEnding;
   bool first = true;
   auto build_dir = build_settings->GetFullPath(build_settings->build_dir())
                        .StripTrailingSeparators();
@@ -253,27 +253,37 @@
         continue;
 
       if (!first) {
-        compile_commands->append(",");
-        compile_commands->append(kPrettyPrintLineEnding);
+        out << ',';
+        out << kPrettyPrintLineEnding;
       }
       first = false;
-      compile_commands->append("  {");
-      compile_commands->append(kPrettyPrintLineEnding);
+      out << "  {";
+      out << kPrettyPrintLineEnding;
 
-      WriteFile(source, path_output, compile_commands);
-      WriteDirectory(base::StringPrintf("%" PRIsFP, PATH_CSTR(build_dir)),
-                     compile_commands);
+      WriteFile(source, path_output, out);
+      WriteDirectory(base::StringPrintf("%" PRIsFP, PATH_CSTR(build_dir)), out);
       WriteCommand(target, source, flags, tool_outputs, path_output,
-                   source_type, tool_name, opts, compile_commands);
-      compile_commands->append("\"");
-      compile_commands->append(kPrettyPrintLineEnding);
-      compile_commands->append("  }");
+                   source_type, tool_name, opts, out);
+      out << "\"";
+      out << kPrettyPrintLineEnding;
+      out << "  }";
     }
   }
 
-  compile_commands->append(kPrettyPrintLineEnding);
-  compile_commands->append("]");
-  compile_commands->append(kPrettyPrintLineEnding);
+  out << kPrettyPrintLineEnding;
+  out << "]";
+  out << kPrettyPrintLineEnding;
+}
+
+}  // namespace
+
+std::string CompileCommandsWriter::RenderJSON(
+    const BuildSettings* build_settings,
+    std::vector<const Target*>& all_targets) {
+  StringOutputBuffer json;
+  std::ostream out(&json);
+  OutputJSON(build_settings, all_targets, out);
+  return json.str();
 }
 
 bool CompileCommandsWriter::RunAndWriteFiles(
@@ -298,17 +308,21 @@
                          base::SPLIT_WANT_NONEMPTY)) {
     target_filters_set.insert(target);
   }
-  std::string json;
+
+  StringOutputBuffer json;
+  std::ostream output_to_json(&json);
   if (target_filters_set.empty()) {
-    RenderJSON(build_settings, all_targets, &json);
+    OutputJSON(build_settings, all_targets, output_to_json);
   } else {
     std::vector<const Target*> preserved_targets =
         FilterTargets(all_targets, target_filters_set);
-    RenderJSON(build_settings, preserved_targets, &json);
+    OutputJSON(build_settings, preserved_targets, output_to_json);
   }
 
-  if (!WriteFileIfChanged(output_path, json, err))
-    return false;
+  if (!json.ContentsEqual(output_path)) {
+    if (!json.WriteToFile(output_path, err))
+      return false;
+  }
   return true;
 }
 
diff --git a/src/gn/compile_commands_writer.h b/src/gn/compile_commands_writer.h
index 78db51e..be4e90b 100644
--- a/src/gn/compile_commands_writer.h
+++ b/src/gn/compile_commands_writer.h
@@ -26,9 +26,10 @@
                                const std::string& target_filters,
                                bool quiet,
                                Err* err);
-  static void RenderJSON(const BuildSettings* build_settings,
-                         std::vector<const Target*>& all_targets,
-                         std::string* compile_commands);
+
+  static std::string RenderJSON(const BuildSettings* build_settings,
+                                std::vector<const Target*>& all_targets);
+
   static std::vector<const Target*> FilterTargets(
       const std::vector<const Target*>& all_targets,
       const std::set<std::string>& target_filters_set);
diff --git a/src/gn/compile_commands_writer_unittest.cc b/src/gn/compile_commands_writer_unittest.cc
index 99dc55d..9bbb076 100644
--- a/src/gn/compile_commands_writer_unittest.cc
+++ b/src/gn/compile_commands_writer_unittest.cc
@@ -54,9 +54,8 @@
 
   // Source set itself.
   {
-    std::string out;
     CompileCommandsWriter writer;
-    writer.RenderJSON(build_settings(), targets, &out);
+    std::string out = writer.RenderJSON(build_settings(), targets);
 
 #if defined(OS_WIN)
     const char expected[] =
@@ -104,9 +103,8 @@
   targets.push_back(&shlib_target);
 
   {
-    std::string out;
     CompileCommandsWriter writer;
-    writer.RenderJSON(build_settings(), targets, &out);
+    std::string out = writer.RenderJSON(build_settings(), targets);
 
 #if defined(OS_WIN)
     const char expected[] =
@@ -166,9 +164,8 @@
   targets.push_back(&stlib_target);
 
   {
-    std::string out;
     CompileCommandsWriter writer;
-    writer.RenderJSON(build_settings(), targets, &out);
+    std::string out = writer.RenderJSON(build_settings(), targets);
 
 #if defined(OS_WIN)
     const char expected[] =
@@ -243,9 +240,8 @@
   ASSERT_TRUE(target.OnResolved(&err));
   targets.push_back(&target);
 
-  std::string out;
   CompileCommandsWriter writer;
-  writer.RenderJSON(build_settings(), targets, &out);
+  std::string out = writer.RenderJSON(build_settings(), targets);
 
   const char expected[] =
       "-DBOOL_DEF -DINT_DEF=123 -DSTR_DEF=\\\\\\\"ABCD\\\\ 1\\\\\\\"";
@@ -302,9 +298,8 @@
     ASSERT_TRUE(no_pch_target.OnResolved(&err));
     targets.push_back(&no_pch_target);
 
-    std::string out;
     CompileCommandsWriter writer;
-    writer.RenderJSON(build_settings(), targets, &out);
+    std::string out = writer.RenderJSON(build_settings(), targets);
 
 #if defined(OS_WIN)
     const char no_pch_expected[] =
@@ -357,9 +352,8 @@
     ASSERT_TRUE(pch_target.OnResolved(&err));
     targets.push_back(&pch_target);
 
-    std::string out;
     CompileCommandsWriter writer;
-    writer.RenderJSON(build_settings(), targets, &out);
+    std::string out = writer.RenderJSON(build_settings(), targets);
 
 #if defined(OS_WIN)
     const char pch_win_expected[] =
@@ -453,9 +447,8 @@
     ASSERT_TRUE(no_pch_target.OnResolved(&err));
     targets.push_back(&no_pch_target);
 
-    std::string out;
     CompileCommandsWriter writer;
-    writer.RenderJSON(build_settings(), targets, &out);
+    std::string out = writer.RenderJSON(build_settings(), targets);
 
 #if defined(OS_WIN)
     const char no_pch_expected[] =
@@ -508,9 +501,8 @@
     ASSERT_TRUE(pch_target.OnResolved(&err));
     targets.push_back(&pch_target);
 
-    std::string out;
     CompileCommandsWriter writer;
-    writer.RenderJSON(build_settings(), targets, &out);
+    std::string out = writer.RenderJSON(build_settings(), targets);
 
 #if defined(OS_WIN)
     const char pch_gcc_expected[] =
@@ -565,9 +557,8 @@
   ASSERT_TRUE(target.OnResolved(&err));
   targets.push_back(&target);
 
-  std::string out;
   CompileCommandsWriter writer;
-  writer.RenderJSON(build_settings(), targets, &out);
+  std::string out = writer.RenderJSON(build_settings(), targets);
 
 #if defined(OS_WIN)
   const char expected[] =
diff --git a/src/gn/json_project_writer.cc b/src/gn/json_project_writer.cc
index c478379..0884aa7 100644
--- a/src/gn/json_project_writer.cc
+++ b/src/gn/json_project_writer.cc
@@ -4,10 +4,17 @@
 
 #include "gn/json_project_writer.h"
 
+#include <algorithm>
+#include <fstream>
 #include <memory>
+#include <unordered_map>
+#include <vector>
 
 #include "base/command_line.h"
+#include "base/files/file_path.h"
+#include "base/files/file_util.h"
 #include "base/json/json_writer.h"
+#include "base/json/string_escape.h"
 #include "base/strings/string_number_conversions.h"
 #include "gn/builder.h"
 #include "gn/commands.h"
@@ -17,6 +24,7 @@
 #include "gn/filesystem_utils.h"
 #include "gn/scheduler.h"
 #include "gn/settings.h"
+#include "gn/string_output_buffer.h"
 
 // Structure of JSON output file
 // {
@@ -150,9 +158,9 @@
     return false;
   }
 
-  std::string json = RenderJSON(build_settings, targets);
-  if (!ContentsEqual(output_path, json)) {
-    if (!WriteFileIfChanged(output_path, json, err)) {
+  StringOutputBuffer json = GenerateJSON(build_settings, targets);
+  if (!json.ContentsEqual(output_path)) {
+    if (!json.WriteToFile(output_path, err)) {
       return false;
     }
 
@@ -177,46 +185,223 @@
   return true;
 }
 
-std::string JSONProjectWriter::RenderJSON(
+namespace {
+
+// NOTE: Intentional macro definition allows compile-time string concatenation.
+// (see usage below).
+#if defined(OS_WINDOWS)
+#define LINE_ENDING "\r\n"
+#else
+#define LINE_ENDING "\n"
+#endif
+
+// Helper class to output a, potentially very large, JSON file to a
+// StringOutputBuffer. Note that sorting the keys, if desired, is left to
+// the user (unlike base::JSONWriter). This allows rendering to be performed
+// in series of incremental steps. Usage is:
+//
+//   1) Create instance, passing a StringOutputBuffer reference as the
+//      destination.
+//
+//   2) Add keys and values using one of the following:
+//
+//       a) AddString(key, string_value) to add one string value.
+//
+//       b) BeginList(key), AddListItem(), EndList() to add a string list.
+//          NOTE: Only lists of strings are supported here.
+//
+//       c) BeginDict(key), ... add other keys, followed by EndDict() to add
+//          a dictionary key.
+//
+//   3) Call Close() or destroy the instance to finalize the output.
+//
+class SimpleJSONWriter {
+ public:
+  // Constructor.
+  SimpleJSONWriter(StringOutputBuffer& out) : out_(out) {
+    out_ << "{" LINE_ENDING;
+    SetIndentation(1u);
+  }
+
+  // Destructor.
+  ~SimpleJSONWriter() { Close(); }
+
+  // Closing finalizes the output.
+  void Close() {
+    if (indentation_ > 0) {
+      DCHECK(indentation_ == 1u);
+      if (comma_.size())
+        out_ << LINE_ENDING;
+
+      out_ << "}" LINE_ENDING;
+      SetIndentation(0);
+    }
+  }
+
+  // Add new string-valued key.
+  void AddString(std::string_view key, std::string_view value) {
+    if (comma_.size()) {
+      out_ << comma_;
+    }
+    AddMargin() << Escape(key) << ": " << Escape(value);
+    comma_ = "," LINE_ENDING;
+  }
+
+  // Begin a new list. Must be followed by zero or more AddListItem() calls,
+  // then by EndList().
+  void BeginList(std::string_view key) {
+    if (comma_.size())
+      out_ << comma_;
+    AddMargin() << Escape(key) << ": [ ";
+    comma_ = {};
+  }
+
+  // Add a new list item. For now only string values are supported.
+  void AddListItem(std::string_view item) {
+    if (comma_.size())
+      out_ << comma_;
+    out_ << Escape(item);
+    comma_ = ", ";
+  }
+
+  // End current list.
+  void EndList() {
+    out_ << " ]";
+    comma_ = "," LINE_ENDING;
+  }
+
+  // Begin new dictionaary. Must be followed by zero or more other key
+  // additions, then a call to EndDict().
+  void BeginDict(std::string_view key) {
+    if (comma_.size())
+      out_ << comma_;
+
+    AddMargin() << Escape(key) << ": {";
+    SetIndentation(indentation_ + 1);
+    comma_ = LINE_ENDING;
+  }
+
+  // End current dictionary.
+  void EndDict() {
+    if (comma_.size())
+      out_ << LINE_ENDING;
+
+    SetIndentation(indentation_ - 1);
+    AddMargin() << "}";
+    comma_ = "," LINE_ENDING;
+  }
+
+  // Add a dictionary-valued key, whose value is already formatted as a valid
+  // JSON string. Useful to insert the output of base::JSONWriter::Write()
+  // into the target buffer.
+  void AddJSONDict(std::string_view key, std::string_view json) {
+    if (comma_.size())
+      out_ << comma_;
+    AddMargin() << Escape(key) << ": ";
+    if (json.empty()) {
+      out_ << "{ }";
+    } else {
+      DCHECK(json[0] == '{');
+      bool first_line = true;
+      do {
+        size_t line_end = json.find('\n');
+
+        // NOTE: Do not add margin if original input line is empty.
+        // This needs to deal with CR/LF which are part of |json| on Windows
+        // only, due to the way base::JSONWriter::Write() is implemented.
+        bool line_empty = (line_end == 0 || (line_end == 1 && json[0] == '\r'));
+        if (!first_line && !line_empty)
+          AddMargin();
+
+        if (line_end == std::string_view::npos) {
+          out_ << json;
+          ;
+          comma_ = {};
+          return;
+        }
+        // Important: do not add the final newline.
+        out_ << json.substr(
+            0, (line_end == json.size() - 1) ? line_end : line_end + 1);
+        json.remove_prefix(line_end + 1);
+        first_line = false;
+      } while (!json.empty());
+    }
+    comma_ = "," LINE_ENDING;
+  }
+
+ private:
+  // Return the JSON-escape version of |str|.
+  static std::string Escape(std::string_view str) {
+    std::string result;
+    base::EscapeJSONString(str, true, &result);
+    return result;
+  }
+
+  // Adjust indentation level.
+  void SetIndentation(size_t indentation) { indentation_ = indentation; }
+
+  // Append margin, and return reference to output buffer.
+  StringOutputBuffer& AddMargin() const {
+    static const char kMargin[17] = "                ";
+    size_t margin_len = indentation_ * 3;
+    while (margin_len > 0) {
+      size_t span = (margin_len > 16u) ? 16u : margin_len;
+      out_.Append(kMargin, span);
+      margin_len -= span;
+    }
+    return out_;
+  }
+
+  size_t indentation_ = 0;
+  std::string_view comma_;
+  StringOutputBuffer& out_;
+};
+
+}  // namespace
+
+StringOutputBuffer JSONProjectWriter::GenerateJSON(
     const BuildSettings* build_settings,
     std::vector<const Target*>& all_targets) {
   Label default_toolchain_label;
+  if (!all_targets.empty())
+    default_toolchain_label =
+        all_targets[0]->settings()->default_toolchain_label();
 
-  auto targets = std::make_unique<base::DictionaryValue>();
-  for (const auto* target : all_targets) {
-    if (default_toolchain_label.is_null())
-      default_toolchain_label = target->settings()->default_toolchain_label();
-    auto description =
-        DescBuilder::DescriptionForTarget(target, "", false, false, false);
-    // Outputs need to be asked for separately.
-    auto outputs = DescBuilder::DescriptionForTarget(target, "source_outputs",
-                                                     false, false, false);
-    base::DictionaryValue* outputs_value = nullptr;
-    if (outputs->GetDictionary("source_outputs", &outputs_value) &&
-        !outputs_value->empty()) {
-      description->MergeDictionary(outputs.get());
-    }
-    targets->SetWithoutPathExpansion(
-        target->label().GetUserVisibleName(default_toolchain_label),
-        std::move(description));
+  StringOutputBuffer out;
+
+  // Sort the targets according to their human visible labels first.
+  std::unordered_map<const Target*, std::string> target_labels;
+  for (const Target* target : all_targets) {
+    target_labels[target] =
+        target->label().GetUserVisibleName(default_toolchain_label);
   }
 
-  auto settings = std::make_unique<base::DictionaryValue>();
-  settings->SetKey("root_path", base::Value(build_settings->root_path_utf8()));
-  settings->SetKey("build_dir",
-                   base::Value(build_settings->build_dir().value()));
-  settings->SetKey(
-      "default_toolchain",
-      base::Value(default_toolchain_label.GetUserVisibleName(false)));
+  std::vector<const Target*> sorted_targets(all_targets.begin(),
+                                            all_targets.end());
+  std::sort(sorted_targets.begin(), sorted_targets.end(),
+            [&target_labels](const Target* a, const Target* b) {
+              return target_labels[a] < target_labels[b];
+            });
 
-  // Other files read by the build.
-  std::vector<base::FilePath> other_files = g_scheduler->GetGenDependencies();
+  SimpleJSONWriter json_writer(out);
 
-  const InputFileManager* input_file_manager =
-      g_scheduler->input_file_manager();
+  // IMPORTANT: Keep the keys sorted when adding them to |json_writer|.
 
-  base::ListValue inputs;
+  json_writer.BeginDict("build_settings");
   {
+    json_writer.AddString("build_dir", build_settings->build_dir().value());
+
+    json_writer.AddString("default_toolchain",
+                          default_toolchain_label.GetUserVisibleName(false));
+
+    json_writer.BeginList("gen_input_files");
+
+    // Other files read by the build.
+    std::vector<base::FilePath> other_files = g_scheduler->GetGenDependencies();
+
+    const InputFileManager* input_file_manager =
+        g_scheduler->input_file_manager();
+
     VectorSetSorter<base::FilePath> sorter(
         input_file_manager->GetInputFileCount() + other_files.size());
 
@@ -225,26 +410,53 @@
     sorter.Add(other_files.begin(), other_files.end());
 
     std::string build_path = FilePathToUTF8(build_settings->root_path());
-    auto item_callback = [&inputs,
+    auto item_callback = [&json_writer,
                           &build_path](const base::FilePath& input_file) {
       std::string file;
       if (MakeAbsolutePathRelativeIfPossible(
               build_path, FilePathToUTF8(input_file), &file)) {
-        inputs.Append(std::make_unique<base::Value>(std::move(file)));
+        json_writer.AddListItem(file);
       }
     };
-
     sorter.IterateOver(item_callback);
+
+    json_writer.EndList();  // gen_input_files
+
+    json_writer.AddString("root_path", build_settings->root_path_utf8());
   }
+  json_writer.EndDict();  // build_settings
 
-  settings->SetKey("gen_input_files", std::move(inputs));
+  json_writer.BeginDict("targets");
+  {
+    for (const auto* target : sorted_targets) {
+      auto description =
+          DescBuilder::DescriptionForTarget(target, "", false, false, false);
+      // Outputs need to be asked for separately.
+      auto outputs = DescBuilder::DescriptionForTarget(target, "source_outputs",
+                                                       false, false, false);
+      base::DictionaryValue* outputs_value = nullptr;
+      if (outputs->GetDictionary("source_outputs", &outputs_value) &&
+          !outputs_value->empty()) {
+        description->MergeDictionary(outputs.get());
+      }
 
-  auto output = std::make_unique<base::DictionaryValue>();
-  output->SetWithoutPathExpansion("targets", std::move(targets));
-  output->SetWithoutPathExpansion("build_settings", std::move(settings));
+      std::string json_dict;
+      base::JSONWriter::WriteWithOptions(*description.get(),
+                                         base::JSONWriter::OPTIONS_PRETTY_PRINT,
+                                         &json_dict);
+      json_writer.AddJSONDict(target_labels[target], json_dict);
+    }
+  }
+  json_writer.EndDict();  // targets
 
-  std::string s;
-  base::JSONWriter::WriteWithOptions(
-      *output.get(), base::JSONWriter::OPTIONS_PRETTY_PRINT, &s);
-  return s;
+  json_writer.Close();
+
+  return out;
+}
+
+std::string JSONProjectWriter::RenderJSON(
+    const BuildSettings* build_settings,
+    std::vector<const Target*>& all_targets) {
+  StringOutputBuffer storage = GenerateJSON(build_settings, all_targets);
+  return storage.str();
 }
diff --git a/src/gn/json_project_writer.h b/src/gn/json_project_writer.h
index 9d396d3..74293a4 100644
--- a/src/gn/json_project_writer.h
+++ b/src/gn/json_project_writer.h
@@ -10,6 +10,7 @@
 
 class Builder;
 class BuildSettings;
+class StringOutputBuffer;
 
 class JSONProjectWriter {
  public:
@@ -27,6 +28,10 @@
   FRIEND_TEST_ALL_PREFIXES(JSONWriter, ForEachWithResponseFile);
   FRIEND_TEST_ALL_PREFIXES(JSONWriter, RustTarget);
 
+  static StringOutputBuffer GenerateJSON(
+      const BuildSettings* build_settings,
+      std::vector<const Target*>& all_targets);
+
   static std::string RenderJSON(const BuildSettings* build_settings,
                                 std::vector<const Target*>& all_targets);
 };
diff --git a/src/gn/rust_project_writer.cc b/src/gn/rust_project_writer.cc
index 754f9b6..19eb426 100644
--- a/src/gn/rust_project_writer.cc
+++ b/src/gn/rust_project_writer.cc
@@ -14,6 +14,7 @@
 #include "gn/ninja_target_command_util.h"
 #include "gn/rust_tool.h"
 #include "gn/source_file.h"
+#include "gn/string_output_buffer.h"
 #include "gn/tool.h"
 
 #if defined(OS_WINDOWS)
@@ -58,15 +59,16 @@
 
   std::vector<const Target*> all_targets = builder.GetAllResolvedTargets();
 
-  std::ofstream json;
-  json.open(FilePathToUTF8(output_path).c_str(),
-            std::ios_base::out | std::ios_base::binary);
-  if (json.fail())
-    return false;
+  StringOutputBuffer out_buffer;
+  std::ostream out(&out_buffer);
 
-  RenderJSON(build_settings, all_targets, json);
+  RenderJSON(build_settings, all_targets, out);
 
-  return true;
+  if (out_buffer.ContentsEqual(output_path)) {
+    return true;
+  }
+
+  return out_buffer.WriteToFile(output_path, err);
 }
 
 using TargetIdxMap = std::unordered_map<const Target*, uint32_t>;
@@ -189,10 +191,10 @@
         rust_project << ",";
       }
       first = false;
-      rust_project << NEWLINE << "        {" NEWLINE
+      rust_project << NEWLINE "        {" NEWLINE
                    << "          \"crate\": " << std::to_string(idx)
-                   << "," NEWLINE << "          \"name\": \"" << dep
-                   << "\"" NEWLINE << "        }";
+                   << "," NEWLINE "          \"name\": \"" << dep
+                   << "\"" NEWLINE "        }";
     }
   }
   rust_project << NEWLINE "      ]," NEWLINE;