| wptrunner Design | |
| ================ | |
| The design of wptrunner is intended to meet the following | |
| requirements: | |
| * Possible to run tests from W3C web-platform-tests. | |
| * Tests should be run as fast as possible. In particular it should | |
| not be necessary to restart the browser between tests, or similar. | |
| * As far as possible, the tests should run in a "normal" browser and | |
| browsing context. In particular many tests assume that they are | |
| running in a top-level browsing context, so we must avoid the use | |
| of an ``iframe`` test container. | |
| * It must be possible to deal with all kinds of behaviour of the | |
| browser runder test, for example, crashing, hanging, etc. | |
| * It should be possible to add support for new platforms and browsers | |
| with minimal code changes. | |
| * It must be possible to run tests in parallel to further improve | |
| performance. | |
| * Test output must be in a machine readable form. | |
| Architecture | |
| ------------ | |
| In order to meet the above requirements, wptrunner is designed to | |
| push as much of the test scheduling as possible into the harness. This | |
| allows the harness to monitor the state of the browser and perform | |
| appropriate action if it gets into an unwanted state e.g. kill the | |
| browser if it appears to be hung. | |
| The harness will typically communicate with the browser via some remote | |
| control protocol such as WebDriver. However for browsers where no such | |
| protocol is supported, other implementation strategies are possible, | |
| typically at the expense of speed. | |
| The overall architecture of wptrunner is shown in the diagram below: | |
| .. image:: architecture.svg | |
| The main entry point to the code is :py:func:`run_tests` in | |
| ``wptrunner.py``. This is responsible for setting up the test | |
| environment, loading the list of tests to be executed, and invoking | |
| the remainder of the code to actually execute some tests. | |
| The test environment is encapsulated in the | |
| :py:class:`TestEnvironment` class. This defers to code in | |
| ``web-platform-tests`` which actually starts the required servers to | |
| run the tests. | |
| The set of tests to run is defined by the | |
| :py:class:`TestLoader`. This is constructed with a | |
| :py:class:`TestFilter` (not shown), which takes any filter arguments | |
| from the command line to restrict the set of tests that will be | |
| run. The :py:class:`TestLoader` reads both the ``web-platform-tests`` | |
| JSON manifest and the expectation data stored in ini files and | |
| produces a :py:class:`multiprocessing.Queue` of tests to run, and | |
| their expected results. | |
| Actually running the tests happens through the | |
| :py:class:`ManagerGroup` object. This takes the :py:class:`Queue` of | |
| tests to be run and starts a :py:class:`testrunner.TestRunnerManager` for each | |
| instance of the browser under test that will be started. These | |
| :py:class:`TestRunnerManager` instances are each started in their own | |
| thread. | |
| A :py:class:`TestRunnerManager` coordinates starting the product under | |
| test, and outputting results from the test. In the case that the test | |
| has timed out or the browser has crashed, it has to restart the | |
| browser to ensure the test run can continue. The functionality for | |
| initialising the browser under test, and probing its state | |
| (e.g. whether the process is still alive) is implemented through a | |
| :py:class:`Browser` object. An implementation of this class must be | |
| provided for each product that is supported. | |
| The functionality for actually running the tests is provided by a | |
| :py:class:`TestRunner` object. :py:class:`TestRunner` instances are | |
| run in their own child process created with the | |
| :py:mod:`multiprocessing` module. This allows them to run concurrently | |
| and to be killed and restarted as required. Communication between the | |
| :py:class:`TestRunnerManager` and the :py:class:`TestRunner` is | |
| provided by a pair of queues, one for sending messages in each | |
| direction. In particular test results are sent from the | |
| :py:class:`TestRunner` to the :py:class:`TestRunnerManager` using one | |
| of these queues. | |
| The :py:class:`TestRunner` object is generic in that the same | |
| :py:class:`TestRunner` is used regardless of the product under | |
| test. However the details of how to run the test may vary greatly with | |
| the product since different products support different remote control | |
| protocols (or none at all). These protocol-specific parts are placed | |
| in the :py:class:`Executor` object. There is typically a different | |
| :py:class:`Executor` class for each combination of control protocol | |
| and test type. The :py:class:`TestRunner` is responsible for pulling | |
| each test off the :py:class:`Queue` of tests and passing it down to | |
| the :py:class:`Executor`. | |
| The executor often requires access to details of the particular | |
| browser instance that it is testing so that it knows e.g. which port | |
| to connect to to send commands to the browser. These details are | |
| encapsulated in the :py:class:`ExecutorBrowser` class. |