# Test Runner Documentation

The scripts in this folder comprise a cross-platform unit test runner. The
components of this runner are as follows:

## Components

### 1.) test_runner.py

This script is used to run unit test binaries on multiple platforms. To see a
full list of parameters that can be supplied to the script, run:

    python test_runner.py --help

#### Running Tests:

To run all tests for a given platform, execute the `test_runner.py` script and
provide at minimum the `--platform` and `--config` arguments.

Example:

	python test_runner.py --platform=android-x86 --config=devel

Running this command will run all unit tests for the `devel` build of the
`android-x86` platform. To specify a device to run the tests on, provide the
`--device_id` argument, as shown below:

    python test_runner.py --platform=android-x86 --config=devel \
      --device_id=emulator-4

#### Running a Single Test:

If you would like to run a single unit test binary and view its output, you can
do so by using the `--target_name` parameter and providing a test binary name,
as shown below:

    python test_runner.py --platform=android-x86 --config=devel \
      --device_id=emulator-4 --target_name=audio_test

#### Building Tests:

You can also use this script to build your unit test binaries before running
them. To do this, provide the `-b` command line flag. If you would like to
build the tests and then run them, provide the flags `-br`.

### 2.) Master list of test binaries

In your application's `starboard_configuration.py` file, define a variable
called TEST_TARGETS. It should be a list containing the names of all of the
test binaries that you want the test runner to run. An example is shown below:

    TEST_TARGETS =[
        'audio_test',
        'bindings_test',
    ]

If your 'starboard_configuration.py' file contains this list, then every
platform you support in Starboard will try to run these test binaries unless
they are filtered out as described below.

## Filtering Tests

To filter out tests that you do not want to run for a specific platform,
implement a method within the platform's `PlatformConfiguration` subclass called
`GetTestFilters()`. The `PlatformConfiguration` subclass lives in the
`gyp_configuration.py` file for each platform. If the tests are
application-specific, you may define `GetTestFilters()` on an optional
`ApplicationConfiguration` subclass, which will be found in the
`<platform-directory>/<application-name>/configuration.py` subdirectory. See
[this Linux implementation](../../linux/x64x11/cobalt/configuration.py) for an
example.

The `GetTestFilters()` function should return a list of `TestFilter` objects,
which can be constructed by importing `starboard.tools.testing.test_filter`. To
make a `TestFilter` object, provide the constructor with the test target name,
the name of the actual unit test that the target runs, and optionally, the build
configuration from which the test should be excluded. An example is shown below:

    test_filter.TestFilter('math_test', 'Vector3dTest.IsZero', 'debug')

If a configuration is not provided, then the test will be exluded from all
configurations.

To filter out all tests for a particular target, provide
`test_filter.FILTER_ALL` as the test name.

To disable unit testing for all targets and all configurations, return a list
containing `test_filter.DISABLE_TESTING`.

## Environment Variables

If a platform requires extra environment variables in order to run tests
properly, implement a method called `GetTestEnvVariables()` in the same
`PlatformConfiguration` or `ApplicationConfiguration` mentioned above. The
application-level variables will be merged on top of the platform-level
variables. There is an example of this method in the provided Linux
implementation. The method should return a dictionary that maps test binary
names to dictionaries of environment variables that they need.

Example:

    def GetTestEnvVariables(self):
      return {
          'base_unittests': {'ASAN_OPTIONS': 'detect_leaks=0'},
          'crypto_unittests': {'ASAN_OPTIONS': 'detect_leaks=0'},
          'net_unittests': {'ASAN_OPTIONS': 'detect_leaks=0'}
      }