Jenkins Build管道和并行作业的问题 [英] Problems with Jenkins Build pipeline and parallel jobs

问题描述

我正在将Jenkins与Build Pipeline插件一起使用来构建一些相当复杂的项目，这些项目需要多个编译步骤:

I am using Jenkins with the Build Pipeline plugin to build some fairly complicated projects that require multiple compilation steps:

1. 构建源RPM.
2. 构建二进制RPM(此操作执行两次，每个平台执行一次).
3. 部署到YUM存储库.

我解决构建要求的策略包括将普通工作分解为参数化的工作，这些工作可以在项目和分支之间重复使用，每个工作代表管道中的一个阶段.每个阶段都由参数触发，并将构建工件传递到管道中的下一个作业.但是，我在使用此策略时遇到了麻烦，确实可以使用一些技巧来尽可能以最优雅，最灵活的方式解决该问题.

My strategy for solving build requirements involves splitting the common work into parameterized jobs that can be reused across projects and branches, with each job representing one stage in the pipeline. Each stage is triggered with parameters, and build artifacts passed along to the next job in the pipeline. However, I'm having some trouble with this strategy, and could really use some tips on how to go about solving this problem in the most elegant and flexible way possible.

更具体地说，有两个公共库，它们由其他项目(但不是所有项目)共享.这些库的构建与从属项目不同，但是在Jenkins中不必指定什么是从属项目.

To be more specific, there are two common libraries, which are shared by other projects (but not all projects). The libraries are built differently from the dependent projects, but it should not be necessary to specify in Jenkins what the dependent projects are.

有多个分支，主分支(每晚重建)，开发分支(轮询更改)，功能分支(也轮询)和发布分支(轮询，但为发布而构建).在多个项目中以相同的方式构建分支.

There are multiple branches, the master branch (rebuilt nightly), the develop branch (polled for changes), feature branches (also polled), and release branches (polled, but built for release). The branches are built the same way across multiple projects.

我们每个月都会创建多个存储库，虽然可以期望为新项目进行一些设置，但通常我们希望它尽可能简单和自动化.

We create multiple repositories every month, and whilst it is feasible to expect a little setup for a new project, generally we want this to be as simple and automated as possible.

1. 我有许多具有多个分支的项目，并且我不希望以相同的方式构建所有分支甚至所有项目.因为大多数构建步骤都是相似的，所以我可以将这些通用步骤转换为参数化的构建作业，并让每个作业触发链中的下一个，沿着链传递参数和构建工件.但是，如果需要跳过其中一个步骤，这会分崩离析，因为我不知道有条件地跳过构建步骤的方法.这意味着我将需要复制构建作业，以便可以为每个管道自定义它们，从而导致大量的构建作业.我可以使用插件的组合来创建作业生成器(例如dsl流，dsl作业等)，并向用户尽可能多地隐藏，但是最完美的Jenkins解决方案是什么?有没有我可能错过的插件或示例?您对此有何经验?

1. I have many projects with multiple branches, and I do not wish to build all branches or even all projects in the same way. Because most of the build steps are similar I can turn these common steps into parameterized build jobs, and get each job to trigger the next in the chain, passing parameters and build artifacts along the chain. However, this falls apart if one of the steps needs to be skipped, because I don't know of a way to conditionally skip a build step. This implies I would need to copy the build jobs so that I can customise them for each pipeline, resulting in a very large number of build jobs. I could use a combination of plugins to create a job generator (eg. dsl flow, dsl job, etc), and hide as much as possible from the users, but what's the most elegant Jenkins solution to this? Are there any plugins, or examples that I might have missed? What's your experience of doing this?

由于步骤2可以分为两个可以并行运行的作业，因此引入了复杂性，这使我的管道出现问题.我的第一次尝试是使用不同的参数触发两次参数化的构建作业，然后再使用join插件将其加入作业，但是从两个上游作业中复制构建工件看起来似乎很复杂.这很重要，因为在阶段3中，我需要这两个作业中的构建工件.结合并行作业并从所有工件中复制工件的最优雅的解决方案是什么?有没有我可能会错过的例子?

Because step 2 can be split into two jobs that can be run in parallel, this introduces a complexity that is causing me problems with my pipeline. My first attempt would trigger a parameterized build job twice with different parameters, and then join the jobs afterwards using the join plugin, but it was beginning to look like it would be complicated to copy in the build artifacts from the two upstream jobs. This is significant, because I need the build artifacts from both jobs for stage 3. What's the most elegant solution to join parallel jobs and copy artifacts from them all? Are there any examples that I might have missed?

我需要合并在阶段2中从两个作业生成的测试结果，并将它们复制到触发构建的作业中.最好的方法是什么?

I need to combine test results generated from both of the jobs in stage 2, and copy them to the job that triggers the build. What's the best way to handle this?

我很高兴阅读文章，演示文稿，技术文章，参考文档，编写脚本以及使该文件正常运行所需的其他任何内容，但我不是Jenkins专家.如果有人可以针对这3个问题给我一些建议，那将很有帮助.此外，如果有相关的建议，我将对如何在Jenkins中充分利用管道CI建设方面的建设性建议表示赞赏.

I'm happy to read articles, presentations, technical articles, reference documentation, write scripts and whatever else necessary to make this work nicely, but I'm not a Jenkins expert. If anyone can give me some advice on these 3 problems then that would be helpful. Additionally, I would appreciate any constructive advice on how to get the best out of pipeline CI builds in Jenkins, if relevant.

推荐答案

关于第一点，我编写的Job Generator插件已针对该用例进行了开发.您可以在Job Generator的Wiki页面上找到更多信息. 还有一种具有不同方法的相同类型的插件(作业生成器作为构建步骤)，它称为

For the first point, the Job Generator plugin I wrote has been developed to address this use case. You can find more info on the wiki page of Job Generator. There is also the same type of plugin with a different approach (job generator as a build step), it is called Jobcopy Builder. The other approaches you mentioned require some kind of DSL and can be a good choice too.

这篇关于Jenkins Build管道和并行作业的问题的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！