Some testing scenarios are intrinsically difficult to automate and require a human to run the test and check the pass condition.
Whenever possible it‘s best to write a fully automated test. For a browser vendor it’s possible to run an automated test hundreds of times a day, but manual tests are likely to be run a handful of times a year. This makes them significantly less useful for catching regressions than automated tests.
However, there are certain scenarios in which this is not yet possible. For example:
Tests that require interaction with browser security UI (e.g. a test in which a user refuses a geolocation permissions grant)
Tests that require interaction with the underlying OS e.g. tests for drag and drop from the desktop onto the browser
Tests that require non-default browser configuration e.g. images disabled
Tests that require interaction with the physical environment e.g. tests that the vibration API causes the device to vibrate or that various sensor APIs respond in the expected way.
There are also some rare cases where it isn't possible to write a layout test as a reftest, and a manual test must be written instead.
Manual tests are distinguished by their filename; all manual tests have filenames of the form name-manual.ext i.e. a -manual suffix after the main filename but before the extension.
Manual tests must be fully self-describing. It is particularly important for these tests that it is easy to determine the result from the information presented to the tester, because a tester may have hundreds of tests to get through, and little understanding of the features that they are testing. Therefore minimalism is a virtue. An ideal self-describing test will have:
Step-by-step instructions for performing the test (if required)
A clear statement of the test outcome (if it can be automatically determined after some setup) or of how to determine the outcome.
Any information other than this (e.g. quotes from the spec) should be avoided.
A convenient way to present the results of a test that can have the result determined by script after some manual setup steps is to use testharness.js to determine and present the result. In this case one must pass {explicit_timeout: true} in a call to setup() in order to disable the automatic timeout of the test. For example:
<!doctype html> <title>Manual click on button triggers onclick handler</title> <script src="/resources/testharness.js"></script> <script src="/resources/testharnessreport.js"></script> <script> setup({explicit_timeout: true}) </script> <p>Click on the button below. If a "PASS" result appears the test passes, otherwise it fails</p> <button onclick="done()">Click Here</button>