詹金斯管道多配置项目 [英] Jenkins Pipeline Multiconfiguration Project
问题描述
原始情况:
我在Jenkins有一份正在运行蚂蚁脚本的工作.我轻松地使用多配置项目"在一个以上的软件版本上测试了该ant脚本.
这种类型的项目真的很酷，因为它允许我指定所需的两个软件的所有版本(在我的情况下为Java和Matlab)，并且它将使用我的所有参数组合来运行我的ant脚本. /p>
这些参数然后用作要在我的蚂蚁使用的可执行文件的位置定义中串联的字符串.
示例:env.MATLAB_EXE =/usr/local/MATLAB/$ {MATLAB_VERSION}/bin/matlab
这工作得很好，但是现在我正在将该脚本迁移到它的基本版本.
管道迁移:
我设法使用参数化管道插件.这样，我达到了可以手动选择将要使用的软件版本的点，如果我手动触发构建，还可以找到一种定期执行此操作的方法，该方法可以在每次运行时选择我想要的参数.
此解决方案似乎相当有效，但并不令人满意.
我的多配置项目具有一些以下功能:
- 我可以设置多个参数以对它们进行插值并执行每种组合
- 执行清晰地分开，并且在构建历史记录/构建详细信息中很容易识别使用了哪些设置
- 只需在参数中添加一个新的可能"值即可产生所需的执行次数
请求
所以我想知道是否有更好的解决方案可以同时满足上述要求.
长话短说:是否有一种方法可以使用詹金斯实现多配置项目，但要使用管道技术?
我已经看到了这个问题，最近又问了很多类似的问题，所以看来解决这个问题很有趣……
以代码可视化的矩阵/多配置作业实际上只是嵌套几个循环，每个参数轴一个.
您可以使用一些经过硬编码的for循环来遍历一些列表来构建相当简单的东西.否则，您可能会变得更加复杂，并进行一些递归循环，因此您不必对特定循环进行硬编码.
免责声明:我做的事情远不止编写代码.我对groovy也很陌生，因此可以更简洁地完成此工作，并且可能有很多groovier事情可以完成，但是无论如何这都能完成工作.
只需做一些工作，此matrixBuilder可以包装在一个类中，这样您就可以传递任务闭包和轴列表并返回任务图.将其粘贴在共享库中，并在任何地方使用.从多配置作业中添加其他一些功能(例如过滤器)应该很容易.
此尝试使用递归matrixBuilder函数遍历任意数量的参数轴并构建所有组合.然后，它并行执行它们(显然取决于节点的可用性).
/*
All the config axes are defined here
Add as many lists of axes in the axisList as you need.
All combinations will be built
*/
def axisList = [
["ubuntu","rhel","windows","osx"], //agents
["jdk6","jdk7","jdk8"], //tools
["banana","apple","orange","pineapple"] //fruit
]
def tasks = [:]
def comboBuilder
def comboEntry = []
def task = {
// builds and returns the task for each combination
/* Map the entries back to a more readable format
the index will correspond to the position of this axis in axisList[] */
def myAgent = it[0]
def myJdk = it[1]
def myFruit = it[2]
return {
// This is where the important work happens for each combination
node(myAgent) {
println "Executing combination ${it.join('-')}"
def javaHome = tool myJdk
println "Node=${env.NODE_NAME}"
println "Java=${javaHome}"
}
//We won't declare a specific agent this part
node {
println "fruit=${myFruit}"
}
}
}
/*
This is where the magic happens
recursively work through the axisList and build all combinations
*/
comboBuilder = { def axes, int level ->
for ( entry in axes[0] ) {
comboEntry[level] = entry
if (axes.size() > 1 ) {
comboBuilder(axes[1..-1], level + 1)
}
else {
tasks[comboEntry.join("-")] = task(comboEntry.collect())
}
}
}
stage ("Setup") {
node {
println "Initial Setup"
}
}
stage ("Setup Combinations") {
node {
comboBuilder(axisList, 0)
}
}
stage ("Multiconfiguration Parallel Tasks") {
//Run the tasks in parallel
parallel tasks
}
stage("The End") {
node {
echo "That's all folks"
}
}
您可以在 http:/上看到更详细的工作流程. /localhost:8080/job/multi-configPipeline/[build]/flowGraphTable/(在构建页面上的管道步骤"链接下提供.
您可以将阶段下移到任务"创建中，然后更清楚地查看每个阶段的详细信息，但不能像多配置作业那样以整洁的矩阵形式查看.
...
return {
// This is where the important work happens for each combination
stage ("${it.join('-')}--build") {
node(myAgent) {
println "Executing combination ${it.join('-')}"
def javaHome = tool myJdk
println "Node=${env.NODE_NAME}"
println "Java=${javaHome}"
}
//Node irrelevant for this part
node {
println "fruit=${myFruit}"
}
}
}
...
或者您可以将每个node用自己的stage包装以获取更多详细信息.
当我这样做时，我注意到我之前的代码中有一个错误(现在已在上面修复).我正在将comboEntry引用传递给任务.我应该发送一份副本，因为尽管阶段的名称正确无误，但在实际执行这些阶段时，这些值当然是所有遇到的最后一个条目.所以我将其更改为tasks[comboEntry.join("-")] = task(comboEntry.collect()).
我注意到您可以在执行并行任务时保留原始的stage ("Multiconfiguration Parallel Tasks") {}.从技术上讲，现在您已嵌套了多个阶段.我不确定詹金斯应该如何处理，但是没有抱怨.但是，父"级时序不包括并行级时序.
我还注意到，当一个新的构建开始运行时，在作业的阶段视图"上，所有以前的构建都消失了，大概是因为阶段名称并没有完全匹配.但是，在构建完成运行之后，它们都再次匹配，并且旧的构建再次出现.
最后，Blue Ocean似乎并没有以同样的方式生动化.它不能识别并行进程中的阶段"，只能识别封闭阶段(如果存在)，如果不能，则不能识别并行".然后仅显示各个并行过程，而不显示其中的各个阶段.
Original situation:
I have a job in Jenkins that is running an ant script. I easily managed to test this ant script on more then one software version using a "Multi-configuration project".
This type of project is really cool because it allows me to specify all the versions of the two software that I need (in my case Java and Matlab) an it will run my ant script with all the combinations of my parameters.
Those parameters are then used as string to be concatenated in the definition of the location of the executable to be used by my ant.
example: env.MATLAB_EXE=/usr/local/MATLAB/${MATLAB_VERSION}/bin/matlab
This is working perfectly but now I am migrating this scripts to a pipline version of it.
Pipeline migration:
I managed to implement the same script in a pipeline fashion using the Parametrized pipelines pluin. With this I achieve the point in which I can manually select which version of my software is going to be used if I trigger the build manually and I also found a way to execute this periodically selecting the parameter I want at each run.
This solution seems fairly working however is not really satisfying.
My multi-config project had some feature that this does not:
- With more then one parameter I can set to interpolate them and execute each combination
- The executions are clearly separated and in build history/build details is easy to recognize which settings hads been used
- Just adding a new "possible" value to the parameter is going to spawn the desired executions
Request
So I wonder if there is a better solution to my problem that can satisfy also the point above.
Long story short: is there a way to implement a multi-configuration project in jenkins but using the pipeline technology?
I've seen this and similar questions asked a lot lately, so it seemed that it would be a fun exercise to work this out...
A matrix/multi-config job, visualized in code, would really just be a few nested for loops, one for each axis of parameters.
You could build something fairly simple with some hard coded for loops to loop over a few lists. Or you can get more complicated and do some recursive looping so you don't have to hard code the specific loops.
DISCLAIMER: I do ops much more than I write code. I am also very new to groovy, so this can probably be done more cleanly, and there are probably a lot of groovier things that could be done, but this gets the job done, anyway.
With a little work, this matrixBuilder could be wrapped up in a class so you could pass in a task closure and the axis list and get the task map back. Stick it in a shared library and use it anywhere. It should be pretty easy to add some of the other features from the multiconfiguration jobs, such as filters.
This attempt uses a recursive matrixBuilder function to work through any number of parameter axes and build all the combinations. Then it executes them in parallel (obviously depending on node availability).
/*
All the config axes are defined here
Add as many lists of axes in the axisList as you need.
All combinations will be built
*/
def axisList = [
["ubuntu","rhel","windows","osx"], //agents
["jdk6","jdk7","jdk8"], //tools
["banana","apple","orange","pineapple"] //fruit
]
def tasks = [:]
def comboBuilder
def comboEntry = []
def task = {
// builds and returns the task for each combination
/* Map the entries back to a more readable format
the index will correspond to the position of this axis in axisList[] */
def myAgent = it[0]
def myJdk = it[1]
def myFruit = it[2]
return {
// This is where the important work happens for each combination
node(myAgent) {
println "Executing combination ${it.join('-')}"
def javaHome = tool myJdk
println "Node=${env.NODE_NAME}"
println "Java=${javaHome}"
}
//We won't declare a specific agent this part
node {
println "fruit=${myFruit}"
}
}
}
/*
This is where the magic happens
recursively work through the axisList and build all combinations
*/
comboBuilder = { def axes, int level ->
for ( entry in axes[0] ) {
comboEntry[level] = entry
if (axes.size() > 1 ) {
comboBuilder(axes[1..-1], level + 1)
}
else {
tasks[comboEntry.join("-")] = task(comboEntry.collect())
}
}
}
stage ("Setup") {
node {
println "Initial Setup"
}
}
stage ("Setup Combinations") {
node {
comboBuilder(axisList, 0)
}
}
stage ("Multiconfiguration Parallel Tasks") {
//Run the tasks in parallel
parallel tasks
}
stage("The End") {
node {
echo "That's all folks"
}
}
You can see a more detailed flow of the job at http://localhost:8080/job/multi-configPipeline/[build]/flowGraphTable/ (available under the Pipeline Steps link on the build page.
EDIT: You can move the stage down into the "task" creation and then see the details of each stage more clearly, but not in a neat matrix like the multi-config job.
...
return {
// This is where the important work happens for each combination
stage ("${it.join('-')}--build") {
node(myAgent) {
println "Executing combination ${it.join('-')}"
def javaHome = tool myJdk
println "Node=${env.NODE_NAME}"
println "Java=${javaHome}"
}
//Node irrelevant for this part
node {
println "fruit=${myFruit}"
}
}
}
...
Or you could wrap each node with their own stage for even more detail.
As I did this, I noticed a bug in my previous code (fixed above now). I was passing the comboEntry reference to the task. I should have sent a copy, because, while the names of the stages were correct, when it actually executed them, the values were, of course, all the last entry encountered. So I changed it to tasks[comboEntry.join("-")] = task(comboEntry.collect()).
I noticed that you can leave the original stage ("Multiconfiguration Parallel Tasks") {} around the execution of the parallel tasks. Technically now you have nested stages. I'm not sure how Jenkins is supposed to handle that, but it doesn't complain. However, the 'parent' stage timing is not inclusive of the parallel stages timing.
I also noticed is that when a new build starts to run, on the "Stage View" of the job, all the previous builds disappear, presumably because the stage names don't all match up. But after the build finishes running, they all match again and the old builds show up again.
And finally, Blue Ocean doesn't seem to vizualize this the same way. It doesn't recognize the "stages" in the parallel processes, only the enclosing stage (if it is present), or "Parallel" if it isn't. And then only shows the individual parallel processes, not the stages within.
这篇关于詹金斯管道多配置项目的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
