如何将脚本作为 pytest 测试运行 [英] How to run script as pytest test

问题描述

假设我有一个测试表示为一个带有 assert 语句的简单脚本(请参阅背景以了解原因)，例如

Suppose I have a test expressed as a simple script with assert-statements (see background for why), e.g

import foo
assert foo(3) == 4

我将如何将此脚本包含在我的 pytest 测试套件中——以一种不错的方式?

How would I include this script in my pytest test suite -- in a nice way?

我尝试了两种有效但不太好的方法:

I have tried two working but less-than-nice approaches:

一种方法是将脚本命名为测试，但这会使整个 pytest 发现在测试失败时失败.

One approach is to name the script like a test, but this makes the whole pytest discovery fail when the test fails.

我目前的方法是从测试函数中导入脚本:

My current approach is to import the script from within a test function:

def test_notebooks():
    notebook_folder = Path(__file__).parent / 'notebooks'
    for notebook in notebook_folder.glob('*.py'):
        import_module(f'{notebook_folder.name}.{notebook.stem}')

这行得通，但脚本不会单独报告，并且测试失败的堆栈跟踪很长且曲折:

This works, but the scripts are not reported individually and test failures have a long and winding stack trace:

__________________________________________________ test_notebooks ___________________________________________________

    def test_notebooks():
        notebook_folder = Path(__file__).parent / 'notebooks'
        for notebook in notebook_folder.glob('*.py'):
>           import_module(f'{notebook_folder.name}.{notebook.stem}')

test_notebooks.py:7:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
envs\anaconda\lib\importlib\__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
<frozen importlib._bootstrap>:1006: in _gcd_import
... (9 lines removed)...
<frozen importlib._bootstrap>:219: in _call_with_frames_removed
    ???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

>   assert False
E   AssertionError

notebooks\notebook_2.py:1: AssertionError

背景

我在脚本文件中进行测试的原因是它们实际上是保存为 .py 文件的 Jupyter 笔记本，并由优秀的 jupytext 插件.

Background

The reason I have test in script files is that they are really Jupyter notebooks saved as .py-files with markup by the excellent jupytext plugin.

这些笔记本被转换为 html 文件，可以交互使用来学习系统，并作为廉价的功能测试.

These notebooks are converted to html for documentation, can be used interactively for learning the system, and serve as cheap functional tests.

推荐答案

在测试函数中执行脚本

从测试函数调用脚本没有任何问题，因此您的方法非常好.但是，我会使用参数化而不是在 for 循环中运行脚本；这样你就可以很好地为每个脚本执行一次测试.如果你不喜欢长的回溯，你可以在自定义 pytest_exception_interact hookimpl.示例:

Executing scripts in a test function

There's nothing wrong with calling the scripts from a test function, so your approach is perfectly fine. However, I would use parametrization instead of running the scripts in a for loop; this way you get the test executed nicely once per script. If you don't like the long tracebacks, you can cut them in a custom pytest_exception_interact hookimpl. Example:

# conftest.py

def pytest_exception_interact(node, call, report):
    excinfo = call.excinfo
    if 'script' in node.funcargs:
        excinfo.traceback = excinfo.traceback.cut(path=node.funcargs['script'])
    report.longrepr = node.repr_failure(excinfo)

参数化测试:

# test_spam.py

import pathlib
import runpy
import pytest

scripts = pathlib.Path(__file__, '..', 'scripts').resolve().glob('*.py')


@pytest.mark.parametrize('script', scripts)
def test_script_execution(script):
    runpy.run_path(script)

测试执行结果(为了测试，我用单行创建了简单的脚本，如 assert False 或 1/0:

Test execution yields (for testing, I have created simple scripts with single lines like assert False or 1 / 0:

$ pytest -v
======================================= test session starts ========================================
platform linux -- Python 3.6.8, pytest-4.6.3, py-1.8.0, pluggy-0.12.0 -- /home/hoefling/projects/.venvs/stackoverflow/bin/python3.6
cachedir: .pytest_cache
rootdir: /home/hoefling/projects/private/stackoverflow/so-56807698
plugins: mock-1.10.4, cov-2.7.1, forked-1.0.2, xdist-1.28.0, django-3.4.8
collected 3 items                                                                                  

test_spam.py::test_script_execution[script0] PASSED
test_spam.py::test_script_execution[script1] FAILED
test_spam.py::test_script_execution[script2] FAILED

============================================= FAILURES =============================================
____________________________________ test_script_runpy[script1] ____________________________________

>   assert False
E   AssertionError

scripts/script_3.py:1: AssertionError
____________________________________ test_script_runpy[script2] ____________________________________

>   1 / 0
E   ZeroDivisionError: division by zero

scripts/script_2.py:1: ZeroDivisionError
================================ 2 failed, 1 passed in 0.07 seconds ================================

自定义测试协议

如果你不喜欢上面的解决方案，我能想到的另一件事是实现你自己的测试集合&执行协议.示例:

Custom test protocol

If you don't like the above solution, another thing I can think of is to implement your own test collection & execution protocol. Example:

# conftest.py

import pathlib
import runpy
import pytest


def pytest_collect_file(parent, path):
    p = pathlib.Path(str(path))
    if p.suffix == '.py' and p.parent.name == 'scripts':
        return Script(path, parent)


class Script(pytest.File):
    def collect(self):
        yield ScriptItem(self.name, self)


class ScriptItem(pytest.Item):
    def runtest(self):
        runpy.run_path(self.fspath)

    def repr_failure(self, excinfo):
        excinfo.traceback = excinfo.traceback.cut(path=self.fspath)
        return super().repr_failure(excinfo)

这将收集 scripts 目录中的每个 .py 文件，将每个脚本包装在一个测试用例中，并在测试执行时调用 runpy.执行日志看起来几乎相同，只是测试命名不同.

This will collect every .py file in scripts directory, wrap each script in a test case and invoke runpy on test execution. The execution log will look pretty much the same, just the tests named differently.

这篇关于如何将脚本作为 pytest 测试运行的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！