使用pytest固定装置一次下载数据，但每次测试均使用新副本 [英] Use a pytest fixture to download data once, but use new copies for each test

问题描述

我有测试代码，可以下载一组图像，对它们进行一些处理，并断言该处理工作符合预期:

  @ pytest.fixturedef image_dir(tmp_path):test_imgs = [#... img URL列表]对于idx，枚举(test_imgs)中的im_url:urllib.request.urlretrieve(im_url，tmp_path/f"{idx} .png")产生tmp_pathdef test_op_A(image_dir:路径):对于image_dir.iterdir()中的im_path:#加载图像#修改图像#将映像保存回磁盘#断言修改工作按预期进行def test_op_B(image_dir:路径):对于image_dir.iterdir()中的im_path:#加载图像#修改图像#将映像保存回磁盘#断言修改工作按预期进行#...更多具有相似格式的测试 

这有效，但是速度非常慢.我怀疑这是因为每次测试都重新下载了图像.

是否有一种干净的方法可以一次创建临时目录，对其进行缓存并为每个测试使用目录的副本?这样，每个测试都可以根据需要修改图像，而不会影响其他测试并保持性能.

解决方案

因此，实现此目标的可能解决方案将使用

现在，我们已将图像缓存在某个位置，我们可以将其夹具复制到每个测试中，而不必下载它们.

from shutil import copytree@ pytest.fixturedef image_dir(tmp_path，request):会话= request.session#将数据从缓存中复制到测试的临时位置复制树(会话.__IMAGE_CACHE，tmp_path)产生tmp_path

I have test code that downloads a set of images, does some processing on them, and asserts that the processing worked as expected:

@pytest.fixture
def image_dir(tmp_path):
    test_imgs = [
        # ... list of img URLs
    ]
    for idx, im_url in enumerate(test_imgs):
        urllib.request.urlretrieve(im_url, tmp_path / f"{idx}.png")
    yield tmp_path

def test_op_A(image_dir: Path):
    for im_path in image_dir.iterdir():
        # load the image
        # modify the image
        # save the image back to disk

    # assert that the modification worked as expected

def test_op_B(image_dir: Path):
    for im_path in image_dir.iterdir():
        # load the image
        # modify the image
        # save the image back to disk

    # assert that the modification worked as expected

# ... more tests with a similar format

This works but is incredibly slow. I suspect that this is because the images are downloaded anew for each test.

Is there a clean way to create the temporary directory once, cache it, and use a copy of the directory for each test? This way each test can modify the images as desired, without influencing the other tests and while remaining performant.

解决方案

So a possible solution to achieve this would make use of pytest_sessionstart and pytest_sessionfinish. We will also use a fixture in order to copy files over.

A general break down of the flow we hope to achieve is the following:

1. Download the necessary files before test collection
2. Have the fixture copy the necessary files to a temporary location
3. After the tests complete, delete the files from the cached location

Place the following two hooks in your conftest.py in the root directory where your tests reside.

from pathlib import Path
from tempfile import TemporaryDirectory

def pytest_sessionstart(session):
    test_imgs = [
        # ... list of img URLs
    ]
    td = TemporaryDirectory()
    tmp_path = Path(td.name)

    for idx, im_url in enumerate(test_imgs):
        urllib.request.urlretrieve(im_url, tmp_path / f"{idx}.png")

    session.__IMAGE_CACHE = tmp_path


def pytest_sessionfinish(session, exitstatus):
    # remove the cached images
    session.__IMAGE_CACHE.cleanup()

Now that we have the images cached in a location, we can have our fixture copy them over for every test instead of having to download them.

from shutil import copytree

@pytest.fixture
def image_dir(tmp_path, request):
    session = request.session
    # copy the data from our cache to the temp location for the test
    copytree(session.__IMAGE_CACHE, tmp_path)
    yield tmp_path

这篇关于使用pytest固定装置一次下载数据，但每次测试均使用新副本的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！