Jenkins-如何处理使用有限资源池的并发作业? [英] Jenkins - How to handle concurrent jobs that use a limited resource pool?

问题描述

我正在尝试改善工作中的某些测试程序，由于我不是詹金斯专家，所以希望你们能为我指明正确的方向吗? 我们目前的情况就是这样.我们有大量定期运行的E2E测试套件.这些测试依赖于有限的资源池(用于运行每个测试的AWS VM).我们有2个测试套件.全面的回归分析在峰值时消耗了约80％的资源，而运行更轻量的烟雾仅使用了15％左右.

I'm trying to improve some of the testing procedures at work and since I'm not an expert on Jenkins was hoping you guys could maybe point me in the right direction?. Our current situation is like this. We have a huge suite of E2E tests that run regularly. These tests rely on a pool of limited resources (AWS VMs that are use to run each tests). We have 2 test suites. A full blown regression that consumes, at its peak, a total of ~80% of those resources and a much more light weight smoke run that just uses 15% or so.

现在，我正在使用可锁定资源插件.当Test Run步骤到来时，它将检查您是否正在运行回归，如果正在运行，则它将请求单次锁定.如果可用，则一切正常，否则，它将一直等到可用之前再继续.这使我可以确保在同一时间点不会运行超过1个回归，但是有很多差距.可能正在运行回归，并且可能会触发多次冒烟，这将耗尽资源池.

Right now I'm using the lockable resources plugin. When the Test Run step comes it checks whether you are running a regression or not and if you are then it will request the single lock. If it is available then all good and if not it will wait until it becomes available before continuing. This allows me to make sure that at no point there will be more than 1 regression running at the same point but it has a lot of gaps. Like a regression could be running and several smoke runs might be triggered which will exhaust the resource pool.

在最理想的情况下，我想完成的是一些条件规则，这些条件规则将基于类似以下的决定来决定测试执行步骤是否可以继续进行:

What I would like to accomplish on a best-case-scenario would be some sort of conditional rules that would decide whether the test execution step can go forward or not based on something like this:

• 一次只能运行1个回归.
• 如果正在运行回归，则仅允许运行1次烟雾运行 平行.
• 如果没有运行回归，则最多允许进行5或6次烟雾测试.
• 如果正在运行2个或更多烟雾测试，则不允许回归 发射.

• Only 1 regression can be running at the same time.
• If a regression is running allow only 1 smoke run to be run in parallel.
• If no regression is running then allow up to 5 or 6 smoke tests.
• If 2 or more smoke tests are running do not allow a regression to launch.

Jenkins管道有可能提供类似的服务吗?在这种情况下，我将使用声明性管道以及我随时间组合在一起的一堆辅助辅助代码.我的第一个想法是查看是否有办法检查可锁定资源是否可用(但尚未实际请求)，然后通过一堆if/then/else设置逻辑.但是同样，我不确定是否有办法检查可锁定资源状态或已经请求了多少种资源.

Would something like that be possible from a Jenkins pipeline? In this case I'm using the declarative pipeline with a bunch of helper groovy code I've put together over time. My first idea is to see if there's a way to check if a lockable resource is available or not (but without actually requesting it yet) and then go through a bunch of if/then/else to set up the logic. But again I'm not sure if there's a way to check a lockable resource state or how many of a kind have already been requested.

老实说，这种复杂的事情可能不在詹金斯应该处理的范围之内，但我不确定并想知道这里是否是一个好的开始.

Honestly, something this complex might probably be outside of what Jenkins is supposed to handle but I'm not sure and figured asking here would be a good start.

谢谢！.

推荐答案

完全忘记更新此内容，但是在阅读并使用lockable resources插件进行了更多尝试之后，我发现您可以在相同的标签和请求下拥有多个资源.特定作业开始时的固定数量.

Completely forgot to update this but after reading and experimenting a bit more with the lockable resources plugin I found out you could have several resources under the same label and request a set quantity whenever a specific job starts.

我定义了5个资源，并设置Jenkinsfile来检查是否正在使用参数regression运行测试套件.如果您运行的是完全回归，它将尝试请求4个锁，而烟雾测试仅尝试请求1个锁.这样，当没有足够的锁可用时，作业将等待，直到有足够的锁可用或超时到期为止.

I defined 5 resources and set the Jenkinsfile to check whether you are running the test suite with the parameter regression or not. If you are running a full regression it will try to request 4 locks while a smoke test will only try to request 1. This way when there aren't enough locks available the job will wait until either the enough amount becomes available or the timeout expires.

这是我的Jenkinsfile中的摘录:

Here's a snippet from my Jenkinsfile:

        stage('Test') {
            steps {
                lock(resource: null, label: 'server-farm-tokens', quantity: getQuantityBySuiteType()) {
                  <<whatever I do to run my tests here>>
                }
            }

由于Jenkin的声明性管道中的错误，

resource必须为空.如果使用脚本化脚本，则可以忽略该参数.

resource has to be null due to a bug in Jenkin's declarative pipeline. If you're using the scripted one you can ignore that parameter.

这篇关于Jenkins-如何处理使用有限资源池的并发作业?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！