blob: 2d8050bd9ca154bdcf8b3a179a7d9d09e5351da5 [file] [log] [blame]
.. highlightlang:: python
.. _`goodpractices`:
Good Integration Practices
.. _`test discovery`:
.. _`Python test discovery`:
Conventions for Python test discovery
``pytest`` implements the following standard test discovery:
* If no arguments are specified then collection starts from :confval:`testpaths`
(if configured) or the current directory. Alternatively, command line arguments
can be used in any combination of directories, file names or node ids.
* recurse into directories, unless they match :confval:`norecursedirs`
* ``test_*.py`` or ``*`` files, imported by their `test package name`_.
* ``Test`` prefixed test classes (without an ``__init__`` method)
* ``test_`` prefixed test functions or methods are test items
For examples of how to customize your test discovery :doc:`example/pythoncollection`.
Within Python modules, ``pytest`` also discovers tests using the standard
:ref:`unittest.TestCase <unittest.TestCase>` subclassing technique.
Choosing a test layout / import rules
``pytest`` supports two common test layouts:
* putting tests into an extra directory outside your actual application
code, useful if you have many functional tests or for other reasons
want to keep tests separate from actual application code (often a good
idea):: # your setuptools Python package metadata
* inlining test directories into your application package, useful if you
have direct relation between (unit-)test and application modules and
want to distribute your tests along with your application:: # your setuptools Python package metadata
Important notes relating to both schemes:
- **make sure that "mypkg" is importable**, for example by typing once::
pip install -e . # install package using in editable mode
- **avoid "" files in your test directories**.
This way your tests can run easily against an installed version
of ``mypkg``, independently from the installed package if it contains
the tests or not.
- With inlined tests you might put ```` into test
directories and make them installable as part of your application.
Using the ``py.test --pyargs mypkg`` invocation pytest will
discover where mypkg is installed and collect tests from there.
With the "external" test you can still distribute tests but they
will not be installed or become importable.
Typically you can run tests by pointing to test directories or modules::
py.test tests/ # for external test dirs
py.test mypkg/test/ # for inlined test dirs
py.test mypkg # run tests in all below test directories
py.test # run all tests below current dir
Because of the above ``editable install`` mode you can change your
source code (both tests and the app) and rerun tests at will.
Once you are done with your work, you can `use tox`_ to make sure
that the package is really correct and tests pass in all
required configurations.
.. note::
You can use Python3 namespace packages (PEP420) for your application
but pytest will still perform `test package name`_ discovery based on the
presence of ```` files. If you use one of the
two recommended file system layouts above but leave away the ````
files from your directories it should just work on Python3.3 and above. From
"inlined tests", however, you will need to use absolute imports for
getting at your application code.
.. _`test package name`:
.. note::
If ``pytest`` finds a "a/b/" test file while
recursing into the filesystem it determines the import name
as follows:
* determine ``basedir``: this is the first "upward" (towards the root)
directory not containing an ````. If e.g. both ``a``
and ``b`` contain an ```` file then the parent directory
of ``a`` will become the ``basedir``.
* perform ``sys.path.insert(0, basedir)`` to make the test module
importable under the fully qualified import name.
* ``import a.b.test_module`` where the path is determined
by converting path separators ``/`` into "." characters. This means
you must follow the convention of having directory and file
names map directly to the import names.
The reason for this somewhat evolved importing technique is
that in larger projects multiple test modules might import
from each other and thus deriving a canonical import name helps
to avoid surprises such as a test modules getting imported twice.
.. _`virtualenv`:
.. _`buildout`:
.. _pip:
.. _`use tox`:
For development, we recommend to use virtualenv_ environments and pip_
for installing your application and any dependencies
as well as the ``pytest`` package itself. This ensures your code and
dependencies are isolated from the system Python installation.
If you frequently release code and want to make sure that your actual
package passes all tests you may want to look into `tox`_, the
virtualenv test automation tool and its `pytest support
Tox helps you to setup virtualenv environments with pre-defined
dependencies and then executing a pre-configured test command with
options. It will run tests against the installed package and not
against your source code checkout, helping to detect packaging
Continuous integration services such as Jenkins_ can make use of the
``--junitxml=PATH`` option to create a JUnitXML file and generate reports.
Integrating with setuptools / ``python test`` / ``pytest-runner``
You can integrate test runs into your setuptools based project
with the `pytest-runner <>`_ plugin.
Add this to ```` file:
.. code-block:: python
from setuptools import setup
setup_requires=['pytest-runner', ...],
tests_require=['pytest', ...],
And create an alias into ``setup.cfg`` file:
.. code-block:: ini
If you now type::
python test
this will execute your tests using ``pytest-runner``. As this is a
standalone version of ``pytest`` no prior installation whatsoever is
required for calling the test command. You can also pass additional
arguments to py.test such as your test directory or other
options using ``--addopts``.
Manual Integration
If for some reason you don't want/can't use ``pytest-runner``, you can write
your own setuptools Test command for invoking pytest.
.. code-block:: python
import sys
from setuptools.command.test import test as TestCommand
class PyTest(TestCommand):
user_options = [('pytest-args=', 'a', "Arguments to pass to py.test")]
def initialize_options(self):
self.pytest_args = []
def run_tests(self):
#import here, cause outside the eggs aren't loaded
import pytest
errno = pytest.main(self.pytest_args)
cmdclass = {'test': PyTest},
Now if you run::
python test
this will download ``pytest`` if needed and then run your tests
as you would expect it to. You can pass a single string of arguments
using the ``--pytest-args`` or ``-a`` command-line option. For example::
python test -a "--durations=5"
is equivalent to running ``py.test --durations=5``.
.. _standalone:
.. _`genscript method`:
(deprecated) Create a pytest standalone script
.. deprecated:: 2.8
.. note::
``genscript`` has been deprecated because:
* It cannot support plugins, rendering its usefulness extremely limited;
* Tooling has become much better since ``genscript`` was introduced;
* It is possible to build a zipped ``pytest`` application without the
shortcomings above.
There's no planned version in which this command will be removed
at the moment of this writing, but its use is discouraged for new
If you are a maintainer or application developer and want people
who don't deal with python much to easily run tests you may generate
a standalone ``pytest`` script::
This generates a ```` script which is a fully functional basic
``pytest`` script, running unchanged under Python2 and Python3.
You can tell people to download the script and then e.g. run it like this::
.. include::