src/starboard/evergreen/testing/README.md

Evergreen Test Automation
=============

The Evergreen test automation reduces the overhead of running the test cases
required to test Evergreen-specific functionality. These tests must be run both
internally and by partners, and making them simple and fast is good for all.

Structure
=============

The Evergreen test automation framework is made up of a few distinct sets of
files.

**Core Scripts**

These are files that are responsible for finding all of the tests, executing all
of the tests, and outputting the results.

* `run_all_tests.sh`
* `setup.sh`
* `pprint.sh`

**Shared Scripts**

These files contain code that is either non-trivial or repeated throughout the
tests, and is shared across all platforms.

* `shared/app_key.sh`
* `shared/drain_file.sh`
* `shared/init_logging.sh`
* `shared/installation_slot.sh`
* `shared/wait_and_watch.sh`

**Platform-Specific Scripts**

These files contain code that is either non-trivial or repeated throughout the
tests, and is platform specific.

* `<PLATFORM>/clean_up.sh`
* `<PLATFORM>/clear_storage.sh`
* `<PLATFORM>/create_file.sh`
* `<PLATFORM>/delete_file.sh`
* `<PLATFORM>/deploy_cobalt.sh`
* `<PLATFORM>/run_command.sh`
* `<PLATFORM>/setup.sh`
* `<PLATFORM>/start_cobalt.sh`
* `<PLATFORM>/stop_cobalt.sh`

**Test HTML**

These files are responsible for changing the channels when tests are running.

* `tests/empty.html`
* `tests/test.html`
* `tests/tseries.html`

**Test Cases**

These files are responsible for the test logic, and each file corresponds to a
single Evergreen test case.

* `tests/abort_update_if_already_updating_test.sh`
* `tests/alternative_content_test.sh`
* `tests/continuous_updates_test.sh`
* `tests/crashing_binary_test.sh`
* `tests/disabled_updater_test.sh`
* `tests/load_slot_being_updated_test.sh`
* `tests/mismatched_architecture_test.sh`
* `tests/noop_binary_test.sh`
* `tests/out_of_storage_test.sh`
* `tests/quick_roll_forward_test.sh`
* `tests/racing_updaters_test.sh`
* `tests/update_fails_verification_test.sh`
* `tests/update_works_for_only_one_app_test.sh`
* `tests/valid_slot_overwritten_test.sh`
* `tests/verify_qa_channel_update_test.sh`

How To Run
=============

Before beginning, please check if your target platform has a README.md and defer
to the steps to run specified there. Otherwise, there are two primary methods of
running the Evergreen tests.

**Python Helper Script**

The Python helper script at `cobalt/tools/buildbot/run_evergreen_tests.py`
simplifies the process of running the automated tests, and relies on the
existing abstract launcher infrastructure.

For this example we will use the following:

* `linux-x64x11` and `qa` for the target platform and configuration.
* `evergreen-x64` and `qa` for the Evergreen platform and configuration.

Then the following command can be used to run the tests.

```
python cobalt/tools/buildbot/run_evergreen_tests.py \
    -p evergreen-x64 -c qa -P linux-x64x11 -C qa
```

**Directly**

First, a directory tree containing the required binaries and content needs to be
created. This directory tree must be in the following format:

Directly running the scripts requires more setup than the helper script above.
We will be using the same platforms and configurations as the steps above.

First, a directory structure needs to be created where the Evergreen binary and
its content is located under the content of the loader binary's content:

```
  linux-x64x11_qa
   +-- deploy
        +-- loader_app
             +-- loader_app <-- loader binary
             +-- content    <-- loader content
                  +-- app
                       +-- cobalt
                            +-- content           <-- cobalt content
                            +-- lib
                                 +-- libcobalt.so <-- cobalt binary
```

Note: This directory structure is the same as what would be generated by
      `starboard/evergreen/shared/launcher.py`.

Next, set the environment variable `OUT` equal to the root of the directory tree
created above.

Then the following command can be used to run the tests.

```
  ./run_all_test.sh linux
```

Tips
=============

The tests will take between 15 and 30 minutes to complete and generate a
significant amount of logs. The tests can be run in the background with the logs
redirected using `./run_all_tests.sh linux &> results &`.

When redirecting all of the logs from the test script being output to `results`,
you can easily check the status by running `grep -E "RUN|PASSED|FAILED"`.

Notes
=============

Evergreen uses "drain" files to ensure only one application downloads an update
at a time. To fake update contention, some tests create this file manually:

* `tests/abort_update_if_already_updating_test.sh`
* `tests/load_slot_being_updated_test.sh`
* `tests/racing_updaters_test.sh`

Evergreen uses "app key" files to keep track of per-application state of an
update. To fake changes to the per-application state of an update, some tests
create, delete, or modify these files manually:

* `tests/update_works_for_only_one_app_test.sh`
* `tests/valid_slot_overwritten_test.sh`

To validate Evergreen behavior when there is not enough storage for an update, a
temporary filesystem is used, only 10MiB in size. One test creates a symbolic
link from the storage path to this filesystem, faking an "out of storage"
situation:

* `tests/out_of_storage_test.sh`