Black Box Test is a place to put End-To-End tests of the Cobalt process. Black Box Tests can launch, send signals to, and terminate Cobalt using platforms' app launcher. Black Box Tests can also control Cobalt using JavaScript or webdriver and can be used to test multiple restarts of the Cobalt process and data that persists between them. They can also be used to test Cobalt's interaction with different web server behavior.
To run all tests:
$ python path/to/black_box_tests.py --platform PLATFORM --config CONFIG
e.g. $ python path/to/black_box_tests.py --platform linux-x64x11 --config devel
To run an individual test:
$ python path/to/black_box_tests.py --platform PLATFORM --config CONFIG --test_name TEST_NAME
e.g. $ python path/to/black_box_tests.py --platform linux-x64x11 --config devel --test_name preload_font
You can apply --gtest_filter through --target_params, e.g. $ python path/to/black_box_tests.py --platform linux-x64x11 --config devel --test_name web_platform_tests --target_params=“--gtest_filter=fetch/*”
Note: When declaring a gtest_filter, it will override the filters defined in GetWebPlatformTestFilters in configuration.py.
If you see a HTTPConnection error which eventually causes a TimeoutException for the test, you might want to add “--server_binding_address localhost” in your python command.
Each script in tests/ includes one python unittest test case. In a test script, you can perform any webdriver operations on Cobalt through BlackBoxCobaltRunner class.
A wrapper around the app launcher. BlackBoxCobaltRunner includes a webdriver module attached to the app launcher's Cobalt instance after it starts running. Includes a method(JSTestsSucceeded()) to check test result on the JavaScript side. Call this method to wait for JavaScript test result. black_box_js_test_utils.js provides some utility functions that are meant to work with runner.JSTestsSucceeded() in the python test scripts. Together, they allow for test logic to exist in either the python test scripts or JavaScript test data. e.g. Call OnEndTest() to signal test completion on the JavaScript side, JSTestsSucceeded() will react to the signal and return the test status of JavaScript test logic; another example is that when the python script wants to wait for some setup steps on JavaScript, call runner.WaitForJSTestsSetup(). Calling setupFinished() in JavaScript whenever ready will unblock the wait.
A default local test server will be launched before any unit test starts to serve the test data in black_box_tests/testdata/. The server's port will be passed to the app launcher to fetch test data from. Test data can include the target web page for Cobalt to open and any additional resources (font file, JavaScripts...). Tests are free to start their own HTTP servers if the default test server is inadequate (e.g. The test is testing that Cobalt handles receipt of a specific HTTP server generated error code properly).