Perfetto Stress Test

This is a test harness that to stress test the client library (DataSource-level only for now).

The test is based on a number of configs in /test/stress_test/configs/*.cfg (NOTE: they must be listed in configs/BUILD.gn). The config is a /protos/perfetto/config/stress_test_config.proto message, which embeds the configuration of the test and a whole trace config.

Each configs defines a testing scenario, determining the general trace config and all the settings of the test (e.g., how many producer processes to spawn, the write timings).

The test is based on exec()-ing traced (the tracing service), perfetto (the consumer cmdline client) and a variable number of stress_producer instances.

stress_producer emits events at a configurable rate, writing predictable sequences of numbers / string, so that the test harness can easily detect corruptions, out-of-order events or gaps.

After running each test, the stress_test binary reads back the trace and performs a bunch of checks:

  • Checks that the number of sequences is exactly equal to #processes x #threads.
  • Checks that each sequence has all the expected packets in the right sequence
  • Checks the payload and correctness of proto nesting of each trace packet.
  • Reports CPU/Memory/Context-switch numbers for the service and producer processes.

Each test config is isolated from the others. All processes are killed and re-spawned for each test.

The stdout/err of each process is saved in a dedicated /tmp/ folder, as well as the resulting trace.

Building and running the test

# This will recursively build traced, perfetto and stress_producer.
ninja -C out/default stress_test

out/default/stress_test

will output:

[307.909] stress_test.cc:116      Saving test results in /tmp/perfetto-ltIBJgA0

===============================================================
Config: simple
===============================================================
Metric               Expected   Actual
------               --------   ------
#Errors              0          0
Duration [ms]        3000       3109
Num threads          1          1
Num packets          1000       1001
Trace size [KB]      168        170
Svc RSS [MB]         4          2
Prod RSS [MB]        ---        1
Svc CPU [ms]         ---        10
Prod CPU [ms]        ---        32
Svc #ctxswitch       ---        103 / 20
Prod #ctxswitch      ---        1022 / 1

===============================================================
Config: bursts
===============================================================
Metric               Expected   Actual
------               --------   ------
#Errors              0          0
Duration [ms]        2000       2381
Num threads          10         10
Num packets          2675       20021
Trace size [KB]      449        11063
Svc RSS [MB]         32         17
Prod RSS [MB]        ---        1
Svc CPU [ms]         ---        98
Prod CPU [ms]        ---        17
Svc #ctxswitch       ---        704 / 1327
Prod #ctxswitch      ---        421 / 1

$ ls -Rlh /tmp/perfetto-ltIBJgA0
total 0
drwxr-xr-x  16 primiano  wheel   512B  5 Aug 09:16 bursts
drwxr-xr-x   9 primiano  wheel   288B  5 Aug 09:16 simple
drwxr-xr-x  38 primiano  wheel   1.2K  5 Aug 09:16 the_storm

/tmp/perfetto-ltIBJgA0/bursts:
total 22752
-rw-r--r--  1 primiano  wheel     0B  5 Aug 09:16 errors.log
-rw-r--r--  1 primiano  wheel   180B  5 Aug 09:16 perfetto.log
-rw-r--r--  1 primiano  wheel   441B  5 Aug 09:16 producer.0.log
...
-rw-r--r--  1 primiano  wheel   441B  5 Aug 09:16 producer.9.log
-rw-------  1 primiano  wheel    11M  5 Aug 09:16 trace
-rw-r--r--  1 primiano  wheel   407B  5 Aug 09:16 traced.log

/tmp/perfetto-ltIBJgA0/simple:
total 400
srwxr-xr-x  1 primiano  wheel     0B  5 Aug 09:16 consumer.sock
-rw-r--r--  1 primiano  wheel     0B  5 Aug 09:16 errors.log
-rw-r--r--  1 primiano  wheel   178B  5 Aug 09:16 perfetto.log
-rw-r--r--  1 primiano  wheel     0B  5 Aug 09:16 producer.0.log
srwxr-xr-x  1 primiano  wheel     0B  5 Aug 09:16 producer.sock
-rw-------  1 primiano  wheel   167K  5 Aug 09:16 trace
-rw-r--r--  1 primiano  wheel   406B  5 Aug 09:16 traced.log

/tmp/perfetto-ltIBJgA0/the_storm:
total 524432
-rw-r--r--  1 primiano  wheel     0B  5 Aug 09:16 errors.log
-rw-r--r--  1 primiano  wheel   184B  5 Aug 09:16 perfetto.log
-rw-r--r--  1 primiano  wheel     0B  5 Aug 09:16 producer.0.log
...
-rw-r--r--  1 primiano  wheel     0B  5 Aug 09:16 producer.127.log
-rw-------  1 primiano  wheel   248M  5 Aug 09:16 trace
-rw-r--r--  1 primiano  wheel   408B  5 Aug 09:16 traced.log

TODOs

The following scenarios requires more coverage:

  • Nested messages.
  • Force losses and check that the last_dropped flag is consistent.
  • Flushes and scraping.
  • Report data losses in the test output.
  • Multibuffer scenarios.
  • write_into_file=true.
  • Vary page size, smb size.