| .. _plugins: |
| .. _`writing-plugins`: |
| |
| Writing plugins |
| =============== |
| |
| It is easy to implement `local conftest plugins`_ for your own project |
| or `pip-installable plugins`_ that can be used throughout many projects, |
| including third party projects. Please refer to :ref:`using plugins` if you |
| only want to use but not write plugins. |
| |
| A plugin contains one or multiple hook functions. :ref:`Writing hooks <writinghooks>` |
| explains the basics and details of how you can write a hook function yourself. |
| ``pytest`` implements all aspects of configuration, collection, running and |
| reporting by calling `well specified hooks`_ of the following plugins: |
| |
| * :ref:`builtin plugins`: loaded from pytest's internal ``_pytest`` directory. |
| |
| * :ref:`external plugins <extplugins>`: modules discovered through |
| `setuptools entry points`_ |
| |
| * `conftest.py plugins`_: modules auto-discovered in test directories |
| |
| In principle, each hook call is a ``1:N`` Python function call where ``N`` is the |
| number of registered implementation functions for a given specification. |
| All specifications and implementations following the ``pytest_`` prefix |
| naming convention, making them easy to distinguish and find. |
| |
| .. _`pluginorder`: |
| |
| Plugin discovery order at tool startup |
| -------------------------------------- |
| |
| ``pytest`` loads plugin modules at tool startup in the following way: |
| |
| * by loading all builtin plugins |
| |
| * by loading all plugins registered through `setuptools entry points`_. |
| |
| * by pre-scanning the command line for the ``-p name`` option |
| and loading the specified plugin before actual command line parsing. |
| |
| * by loading all :file:`conftest.py` files as inferred by the command line |
| invocation: |
| |
| - if no test paths are specified use current dir as a test path |
| - if exists, load ``conftest.py`` and ``test*/conftest.py`` relative |
| to the directory part of the first test path. |
| |
| Note that pytest does not find ``conftest.py`` files in deeper nested |
| sub directories at tool startup. It is usually a good idea to keep |
| your conftest.py file in the top level test or project root directory. |
| |
| * by recursively loading all plugins specified by the |
| ``pytest_plugins`` variable in ``conftest.py`` files |
| |
| |
| .. _`pytest/plugin`: http://bitbucket.org/pytest-dev/pytest/src/tip/pytest/plugin/ |
| .. _`conftest.py plugins`: |
| .. _`conftest.py`: |
| .. _`localplugin`: |
| .. _`conftest`: |
| .. _`local conftest plugins`: |
| |
| conftest.py: local per-directory plugins |
| ---------------------------------------- |
| |
| Local ``conftest.py`` plugins contain directory-specific hook |
| implementations. Hook Session and test running activities will |
| invoke all hooks defined in ``conftest.py`` files closer to the |
| root of the filesystem. Example of implementing the |
| ``pytest_runtest_setup`` hook so that is called for tests in the ``a`` |
| sub directory but not for other directories:: |
| |
| a/conftest.py: |
| def pytest_runtest_setup(item): |
| # called for running each test in 'a' directory |
| print ("setting up", item) |
| |
| a/test_sub.py: |
| def test_sub(): |
| pass |
| |
| test_flat.py: |
| def test_flat(): |
| pass |
| |
| Here is how you might run it:: |
| |
| py.test test_flat.py # will not show "setting up" |
| py.test a/test_sub.py # will show "setting up" |
| |
| .. Note:: |
| If you have ``conftest.py`` files which do not reside in a |
| python package directory (i.e. one containing an ``__init__.py``) then |
| "import conftest" can be ambiguous because there might be other |
| ``conftest.py`` files as well on your PYTHONPATH or ``sys.path``. |
| It is thus good practice for projects to either put ``conftest.py`` |
| under a package scope or to never import anything from a |
| conftest.py file. |
| |
| |
| Writing your own plugin |
| ----------------------- |
| |
| .. _`setuptools`: http://pypi.python.org/pypi/setuptools |
| |
| If you want to write a plugin, there are many real-life examples |
| you can copy from: |
| |
| * a custom collection example plugin: :ref:`yaml plugin` |
| * around 20 :ref:`builtin plugins` which provide pytest's own functionality |
| * many `external plugins <http://plugincompat.herokuapp.com>`_ providing additional features |
| |
| All of these plugins implement the documented `well specified hooks`_ |
| to extend and add functionality. |
| |
| .. note:: |
| Make sure to check out the excellent |
| `cookiecutter-pytest-plugin <https://github.com/pytest-dev/cookiecutter-pytest-plugin>`_ |
| project, which is a `cookiecutter template <https://github.com/audreyr/cookiecutter>`_ |
| for authoring plugins. |
| |
| The template provides an excellent starting point with a working plugin, |
| tests running with tox, comprehensive README and |
| entry-pointy already pre-configured. |
| |
| Also consider :ref:`contributing your plugin to pytest-dev<submitplugin>` |
| once it has some happy users other than yourself. |
| |
| |
| .. _`setuptools entry points`: |
| .. _`pip-installable plugins`: |
| |
| Making your plugin installable by others |
| ---------------------------------------- |
| |
| If you want to make your plugin externally available, you |
| may define a so-called entry point for your distribution so |
| that ``pytest`` finds your plugin module. Entry points are |
| a feature that is provided by `setuptools`_. pytest looks up |
| the ``pytest11`` entrypoint to discover its |
| plugins and you can thus make your plugin available by defining |
| it in your setuptools-invocation: |
| |
| .. sourcecode:: python |
| |
| # sample ./setup.py file |
| from setuptools import setup |
| |
| setup( |
| name="myproject", |
| packages = ['myproject'] |
| |
| # the following makes a plugin available to pytest |
| entry_points = { |
| 'pytest11': [ |
| 'name_of_plugin = myproject.pluginmodule', |
| ] |
| }, |
| ) |
| |
| If a package is installed this way, ``pytest`` will load |
| ``myproject.pluginmodule`` as a plugin which can define |
| `well specified hooks`_. |
| |
| |
| |
| |
| Requiring/Loading plugins in a test module or conftest file |
| ----------------------------------------------------------- |
| |
| You can require plugins in a test module or a conftest file like this:: |
| |
| pytest_plugins = "name1", "name2", |
| |
| When the test module or conftest plugin is loaded the specified plugins |
| will be loaded as well. You can also use dotted path like this:: |
| |
| pytest_plugins = "myapp.testsupport.myplugin" |
| |
| which will import the specified module as a ``pytest`` plugin. |
| |
| |
| Accessing another plugin by name |
| -------------------------------- |
| |
| If a plugin wants to collaborate with code from |
| another plugin it can obtain a reference through |
| the plugin manager like this: |
| |
| .. sourcecode:: python |
| |
| plugin = config.pluginmanager.getplugin("name_of_plugin") |
| |
| If you want to look at the names of existing plugins, use |
| the ``--traceconfig`` option. |
| |
| Testing plugins |
| --------------- |
| |
| pytest comes with some facilities that you can enable for testing your |
| plugin. Given that you have an installed plugin you can enable the |
| :py:class:`testdir <_pytest.pytester.Testdir>` fixture via specifying a |
| command line option to include the pytester plugin (``-p pytester``) or |
| by putting ``pytest_plugins = "pytester"`` into your test or |
| ``conftest.py`` file. You then will have a ``testdir`` fixture which you |
| can use like this:: |
| |
| # content of test_myplugin.py |
| |
| pytest_plugins = "pytester" # to get testdir fixture |
| |
| def test_myplugin(testdir): |
| testdir.makepyfile(""" |
| def test_example(): |
| pass |
| """) |
| result = testdir.runpytest("--verbose") |
| result.fnmatch_lines(""" |
| test_example* |
| """) |
| |
| Note that by default ``testdir.runpytest()`` will perform a pytest |
| in-process. You can pass the command line option ``--runpytest=subprocess`` |
| to have it happen in a subprocess. |
| |
| Also see the :py:class:`RunResult <_pytest.pytester.RunResult>` for more |
| methods of the result object that you get from a call to ``runpytest``. |
| |
| .. _`writinghooks`: |
| |
| Writing hook functions |
| ====================== |
| |
| |
| .. _validation: |
| |
| hook function validation and execution |
| -------------------------------------- |
| |
| pytest calls hook functions from registered plugins for any |
| given hook specification. Let's look at a typical hook function |
| for the ``pytest_collection_modifyitems(session, config, |
| items)`` hook which pytest calls after collection of all test items is |
| completed. |
| |
| When we implement a ``pytest_collection_modifyitems`` function in our plugin |
| pytest will during registration verify that you use argument |
| names which match the specification and bail out if not. |
| |
| Let's look at a possible implementation: |
| |
| .. code-block:: python |
| |
| def pytest_collection_modifyitems(config, items): |
| # called after collection is completed |
| # you can modify the ``items`` list |
| |
| Here, ``pytest`` will pass in ``config`` (the pytest config object) |
| and ``items`` (the list of collected test items) but will not pass |
| in the ``session`` argument because we didn't list it in the function |
| signature. This dynamic "pruning" of arguments allows ``pytest`` to |
| be "future-compatible": we can introduce new hook named parameters without |
| breaking the signatures of existing hook implementations. It is one of |
| the reasons for the general long-lived compatibility of pytest plugins. |
| |
| Note that hook functions other than ``pytest_runtest_*`` are not |
| allowed to raise exceptions. Doing so will break the pytest run. |
| |
| |
| |
| firstresult: stop at first non-None result |
| ------------------------------------------- |
| |
| Most calls to ``pytest`` hooks result in a **list of results** which contains |
| all non-None results of the called hook functions. |
| |
| Some hook specifications use the ``firstresult=True`` option so that the hook |
| call only executes until the first of N registered functions returns a |
| non-None result which is then taken as result of the overall hook call. |
| The remaining hook functions will not be called in this case. |
| |
| |
| hookwrapper: executing around other hooks |
| ------------------------------------------------- |
| |
| .. currentmodule:: _pytest.core |
| |
| .. versionadded:: 2.7 (experimental) |
| |
| pytest plugins can implement hook wrappers which wrap the execution |
| of other hook implementations. A hook wrapper is a generator function |
| which yields exactly once. When pytest invokes hooks it first executes |
| hook wrappers and passes the same arguments as to the regular hooks. |
| |
| At the yield point of the hook wrapper pytest will execute the next hook |
| implementations and return their result to the yield point in the form of |
| a :py:class:`CallOutcome` instance which encapsulates a result or |
| exception info. The yield point itself will thus typically not raise |
| exceptions (unless there are bugs). |
| |
| Here is an example definition of a hook wrapper:: |
| |
| import pytest |
| |
| @pytest.hookimpl(hookwrapper=True) |
| def pytest_pyfunc_call(pyfuncitem): |
| # do whatever you want before the next hook executes |
| |
| outcome = yield |
| # outcome.excinfo may be None or a (cls, val, tb) tuple |
| |
| res = outcome.get_result() # will raise if outcome was exception |
| # postprocess result |
| |
| Note that hook wrappers don't return results themselves, they merely |
| perform tracing or other side effects around the actual hook implementations. |
| If the result of the underlying hook is a mutable object, they may modify |
| that result but it's probably better to avoid it. |
| |
| |
| Hook function ordering / call example |
| ------------------------------------- |
| |
| For any given hook specification there may be more than one |
| implementation and we thus generally view ``hook`` execution as a |
| ``1:N`` function call where ``N`` is the number of registered functions. |
| There are ways to influence if a hook implementation comes before or |
| after others, i.e. the position in the ``N``-sized list of functions: |
| |
| .. code-block:: python |
| |
| # Plugin 1 |
| @pytest.hookimpl(tryfirst=True) |
| def pytest_collection_modifyitems(items): |
| # will execute as early as possible |
| |
| # Plugin 2 |
| @pytest.hookimpl(trylast=True) |
| def pytest_collection_modifyitems(items): |
| # will execute as late as possible |
| |
| # Plugin 3 |
| @pytest.hookimpl(hookwrapper=True) |
| def pytest_collection_modifyitems(items): |
| # will execute even before the tryfirst one above! |
| outcome = yield |
| # will execute after all non-hookwrappers executed |
| |
| Here is the order of execution: |
| |
| 1. Plugin3's pytest_collection_modifyitems called until the yield point |
| because it is a hook wrapper. |
| |
| 2. Plugin1's pytest_collection_modifyitems is called because it is marked |
| with ``tryfirst=True``. |
| |
| 3. Plugin2's pytest_collection_modifyitems is called because it is marked |
| with ``trylast=True`` (but even without this mark it would come after |
| Plugin1). |
| |
| 4. Plugin3's pytest_collection_modifyitems then executing the code after the yield |
| point. The yield receives a :py:class:`CallOutcome` instance which encapsulates |
| the result from calling the non-wrappers. Wrappers shall not modify the result. |
| |
| It's possible to use ``tryfirst`` and ``trylast`` also in conjunction with |
| ``hookwrapper=True`` in which case it will influence the ordering of hookwrappers |
| among each other. |
| |
| |
| Declaring new hooks |
| ------------------------ |
| |
| .. currentmodule:: _pytest.hookspec |
| |
| Plugins and ``conftest.py`` files may declare new hooks that can then be |
| implemented by other plugins in order to alter behaviour or interact with |
| the new plugin: |
| |
| .. autofunction:: pytest_addhooks |
| |
| Hooks are usually declared as do-nothing functions that contain only |
| documentation describing when the hook will be called and what return values |
| are expected. |
| |
| For an example, see `newhooks.py`_ from :ref:`xdist`. |
| |
| .. _`newhooks.py`: https://github.com/pytest-dev/pytest-xdist/blob/974bd566c599dc6a9ea291838c6f226197208b46/xdist/newhooks.py |
| |
| |
| Optionally using hooks from 3rd party plugins |
| --------------------------------------------- |
| |
| Using new hooks from plugins as explained above might be a little tricky |
| because of the standard :ref:`validation mechanism <validation>`: |
| if you depend on a plugin that is not installed, validation will fail and |
| the error message will not make much sense to your users. |
| |
| One approach is to defer the hook implementation to a new plugin instead of |
| declaring the hook functions directly in your plugin module, for example:: |
| |
| # contents of myplugin.py |
| |
| class DeferPlugin(object): |
| """Simple plugin to defer pytest-xdist hook functions.""" |
| |
| def pytest_testnodedown(self, node, error): |
| """standard xdist hook function. |
| """ |
| |
| def pytest_configure(config): |
| if config.pluginmanager.hasplugin('xdist'): |
| config.pluginmanager.register(DeferPlugin()) |
| |
| This has the added benefit of allowing you to conditionally install hooks |
| depending on which plugins are installed. |
| |
| .. _`well specified hooks`: |
| |
| .. currentmodule:: _pytest.hookspec |
| |
| pytest hook reference |
| ===================== |
| |
| |
| Initialization, command line and configuration hooks |
| ---------------------------------------------------- |
| |
| .. autofunction:: pytest_load_initial_conftests |
| .. autofunction:: pytest_cmdline_preparse |
| .. autofunction:: pytest_cmdline_parse |
| .. autofunction:: pytest_namespace |
| .. autofunction:: pytest_addoption |
| .. autofunction:: pytest_cmdline_main |
| .. autofunction:: pytest_configure |
| .. autofunction:: pytest_unconfigure |
| |
| Generic "runtest" hooks |
| ----------------------- |
| |
| All runtest related hooks receive a :py:class:`pytest.Item` object. |
| |
| .. autofunction:: pytest_runtest_protocol |
| .. autofunction:: pytest_runtest_setup |
| .. autofunction:: pytest_runtest_call |
| .. autofunction:: pytest_runtest_teardown |
| .. autofunction:: pytest_runtest_makereport |
| |
| For deeper understanding you may look at the default implementation of |
| these hooks in :py:mod:`_pytest.runner` and maybe also |
| in :py:mod:`_pytest.pdb` which interacts with :py:mod:`_pytest.capture` |
| and its input/output capturing in order to immediately drop |
| into interactive debugging when a test failure occurs. |
| |
| The :py:mod:`_pytest.terminal` reported specifically uses |
| the reporting hook to print information about a test run. |
| |
| Collection hooks |
| ---------------- |
| |
| ``pytest`` calls the following hooks for collecting files and directories: |
| |
| .. autofunction:: pytest_ignore_collect |
| .. autofunction:: pytest_collect_directory |
| .. autofunction:: pytest_collect_file |
| |
| For influencing the collection of objects in Python modules |
| you can use the following hook: |
| |
| .. autofunction:: pytest_pycollect_makeitem |
| .. autofunction:: pytest_generate_tests |
| |
| After collection is complete, you can modify the order of |
| items, delete or otherwise amend the test items: |
| |
| .. autofunction:: pytest_collection_modifyitems |
| |
| Reporting hooks |
| --------------- |
| |
| Session related reporting hooks: |
| |
| .. autofunction:: pytest_collectstart |
| .. autofunction:: pytest_itemcollected |
| .. autofunction:: pytest_collectreport |
| .. autofunction:: pytest_deselected |
| .. autofunction:: pytest_report_header |
| .. autofunction:: pytest_report_teststatus |
| .. autofunction:: pytest_terminal_summary |
| |
| And here is the central hook for reporting about |
| test execution: |
| |
| .. autofunction:: pytest_runtest_logreport |
| |
| You can also use this hook to customize assertion representation for some |
| types: |
| |
| .. autofunction:: pytest_assertrepr_compare |
| |
| |
| Debugging/Interaction hooks |
| --------------------------- |
| |
| There are few hooks which can be used for special |
| reporting or interaction with exceptions: |
| |
| .. autofunction:: pytest_internalerror |
| .. autofunction:: pytest_keyboard_interrupt |
| .. autofunction:: pytest_exception_interact |
| .. autofunction:: pytest_enter_pdb |
| |
| |
| Reference of objects involved in hooks |
| ====================================== |
| |
| .. autoclass:: _pytest.config.Config() |
| :members: |
| |
| .. autoclass:: _pytest.config.Parser() |
| :members: |
| |
| .. autoclass:: _pytest.main.Node() |
| :members: |
| |
| .. autoclass:: _pytest.main.Collector() |
| :members: |
| :show-inheritance: |
| |
| .. autoclass:: _pytest.main.Item() |
| :members: |
| :show-inheritance: |
| |
| .. autoclass:: _pytest.python.Module() |
| :members: |
| :show-inheritance: |
| |
| .. autoclass:: _pytest.python.Class() |
| :members: |
| :show-inheritance: |
| |
| .. autoclass:: _pytest.python.Function() |
| :members: |
| :show-inheritance: |
| |
| .. autoclass:: _pytest.runner.CallInfo() |
| :members: |
| |
| .. autoclass:: _pytest.runner.TestReport() |
| :members: |
| |
| .. autoclass:: _pytest.vendored_packages.pluggy._CallOutcome() |
| :members: |
| |
| .. autofunction:: _pytest.config.get_plugin_manager() |
| |
| .. autoclass:: _pytest.config.PytestPluginManager() |
| :members: |
| :undoc-members: |
| :show-inheritance: |
| |
| .. autoclass:: pluggy.PluginManager() |
| :members: |
| |
| .. currentmodule:: _pytest.pytester |
| |
| .. autoclass:: Testdir() |
| :members: runpytest,runpytest_subprocess,runpytest_inprocess,makeconftest,makepyfile |
| |
| .. autoclass:: RunResult() |
| :members: |
| |
| .. autoclass:: LineMatcher() |
| :members: |