使用pytest动态控制测试顺序 [英] Dynamically control order of tests with pytest
问题描述
我想使用逻辑来控制测试的顺序，这些逻辑将在测试已经运行时即时对其进行排序.
I would like to control the order of my tests using logic which will reorder them on the fly, while they are already running.
我的用例是这样的:我将我的测试与xdist并行化，并且每个测试都使用来自公共池和有限池的外部资源.有些测试比其他测试使用更多的资源，因此，在任何给定时间，只有一部分资源可用时，有些测试具有运行所需的资源，而有些则不需要.
My use case is this: I am parallelizing my tests with xdist, and each test uses external resources from a common and limited pool. Some tests use more resources than others, so at any given time when only a fraction of the resources are available, some of the tests have the resources they need to run and others don't.
我想优化资源的使用，因此我想根据当前可用资源动态选择下一步运行的测试.我会在收集阶段计算出最佳排序，但是我不预先知道每个测试将花费多长时间，因此我无法预测何时可以使用哪些资源.
I want to optimize the usage of the resources, so I would like to dynamically choose which test will run next, based on the resources currently available. I would calculate an optimal ordering during the collection stage, but I don't know in advance how long each test will take, so I can't predict which resources will be available when.
我还没有找到一种使用插件来实现此行为的方法，因为收集阶段似乎与运行阶段不同，并且我不知道另一种方法来修改测试列表以运行除收集之外的其他方法钩子.
I haven't found a way to implement this behavior with a plugin because the collection stage seems to be distinct from the running stage, and I don't know another way to modify the list of tests to run other than the collection hooks.
非常感谢任何建议，无论是实施此建议的方法还是可以优化资源利用的替代方案.我的替代方法是编写自己的简单测试运行程序，但我不想放弃pytest提供的其余功能.
Would very much appreciate any suggestions, whether a way to implement this or an alternative idea which would optimize the resources utilization. My alternative is to write my own simplistic test runner, but I don't want to give up on the rest of what pytest offers.
谢谢！
推荐答案
不是您问题的完整答案，而是有关pytest的问题和一些提示.
Not really an full answer to your problem, but to your question on pytest, and few hints.
要更改pytest中的测试顺序(仅pytest，而不是pytest-xdist)，您可以通过安装以下挂钩包装程序随时更改测试项的顺序:
To change the order of the tests in pytest (just pytest, not pytest-xdist), you can just alter the order of the test items on the go by installing this hook wrapper:
conftest.py:
import pytest
import random
random.seed()
@pytest.hookimpl(hookwrapper=True)
def pytest_runtest_protocol(item, nextitem):
yield
idx = item.session.items.index(item)
remaining = item.session.items[idx+1:]
random.shuffle(remaining)
item.session.items[idx+1:] = remaining
更改已经执行的内容没有任何意义，而仅更改剩余的内容，因此，[idx+1:].在此示例中，我只是对项目进行了混洗，但是您可以对其余功能的列表进行任何操作.
It makes no sense to change what was already executed, but only what remains — hence, [idx+1:]. In this example, I just shuffle the items, but you can do whatever you want with the list of the remaining features.
请紧记:这可能会影响灯具的设置和安装方式.拆解(范围在功能"上方的那些).最初，pytest订购测试，以便他们可以以最有效的方式利用固定装置.特别是，nextitem参数在内部用于跟踪是否应完成治具.由于您每次都有效地更改它，因此效果可能是不可预测的.
Keep in mind: This can affect how the fixtures are setup'ed & teardown'ed (those of scope above 'function'). Originally, pytest orders the tests so they can utilise the fixtures in a most efficient way. And specifically, the nextitem argument is used internally to track if the fixture should be finished. Since you efficiently change it every time, the effects can be unpredictable.
使用pytest-xdist，一切都变得完全不同.您必须阅读 pytest-dist如何在从站之间分布测试.
With pytest-xdist, it all goes totally different. You have to read on how pytest-dist distributes the tests across the slaves.
首先，每个从站都以相同的顺序收集所有相同的测试.如果顺序不同，则pytest-xdist将失败.因此，您无法在收藏时对它们进行重新排序.
First, every slave collects all the same tests and exactly in the same order. If the order is different, pytest-xdist will fail. So you cannot reorder them on collection.
第二，主进程发送该收集列表中的测试索引，作为下一个要执行的测试.因此，集合必须始终保持不变.
Second, the master process sends the test indexes in that collected list as the next tests to execute. So, the collection must be unchanged all the time.
第三，您可以重新定义 schedule()方法调度测试逻辑.
Third, you can redefine the pytest_xdist_make_scheduler hook. There are few sample implementations in the pytest-xdist itself. You can define your own logic of scheduling the tests in schedule() method, using the nodes added/removed and tests collected via the corresponding methods.
第四，那太容易了. .schedule()方法仅在
Fourth, that would be too easy. The .schedule() method is called only in slave_collectionfinish event sent from the slave. I'm sad to say this, but you will have to kill and restart the slave processes all the time to trigger this event, and to re-schedule the remaining tests.
如您所见，pytest-xdist的实现将是巨大的&复杂的.但是，我希望这会给您一些提示，以帮助您了解.
As you can see, the pytest-xdist's implementation will be huge & complex. But I hope this will give you some hints where to look at.
这篇关于使用pytest动态控制测试顺序的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
