在 CI/CD 中如何管理前端和后端之间的依赖关系? [英] In CI/CD how to manage dependency between frontend and backend?

问题描述

我将描述我的设置以使问题不那么抽象，但它们似乎并不针对我的案例.

上下文

我们有 Python-Django 后端和一个 VueJS 前端，每个都在一个存储库中，使用 Portainer(使用堆栈)配置和部署 Gitlab-CI.每个存储库的生产分支中的提交遵循以下路径:

1. 提交
2. gitlab-ci 管道:
   1. 构建 docker 镜像
   2. 测试图像(前端针对已部署的后端进行测试)
   3. 将图片标记为生产:最新
   4. 将图像推回 gitlab 注册表
   5. 在 portainer 中 webhook 对应的服务(前端/后端)来更新部署的镜像

3. 搬运工:
   1. 拉图片
   2. 部署

问题

部署同步

假设我们正在对前端和后端进行重大更改，并且两者都将与以前的版本不兼容.所以新版本必须同时部署.

在我们当前的设置中，我们必须首先部署后端(这会破坏已部署的前端)，然后部署新的前端，修复生产，但有一个停机"期.

测试的分支依赖

有时我们在前端开发分支 feature-1 时，必须针对后端的分支 feature-1 进行测试.

在我们当前的设置中，前端的所有提交都针对已部署的后端进行测试(为避免在 CI 中复制后端，仅使用生产 API 地址)，在这种情况下会导致错误的测试结果.

后端集成测试

当提交到后端时，它可能会破坏前端.

目前后端未针对前端进行测试(仅另一种方式).

可能的解决方案

对于部署同步问题，我考虑创建另一个存储库，其中只有一个文件指定应部署的前端和后端版本.此存储库中的提交将导致 Portanier 的两个服务 webhook 被卷曲"以进行更新(后端和前端).这并不能保证同时更新(在 Portainer 中可能会失败并且不会回滚)，但它会比当前设置更好.

我不确定这里应该使用什么来指定版本:commit hash、git tag、branch、docker image version... 最后一个可能避免重建和测试图像，但我认为图像名称和版本在 Portainer 的堆栈定义中是固定的，不容易自动更新.

对于分支依赖性测试，我考虑在每个存储库(前端和后端)中都有一个文件，指定要测试后端/前端的哪个分支.但是每个存储库的 CI 必须复制整个部署环境(例如，运行一个新的后端和前端来测试每个前端提交).这也将允许后端集成测试.由于我们使用的是 Docker，这并不是很复杂，但是每个 CI 管道都会花费额外的时间......此外，当第一个存储库(前端或后端)被提交时，它将引用另一个仍然不存在的分支存储库，然后失败...

这些解决方案对我来说似乎很尴尬，特别是如果这些是使用 Docker 的 CI/CD 常见的问题.当我们向组合中添加更多存储库时，它会变得更加丑陋.

替代方案?

感谢关注！

(edit: 出于好奇，我目前的设置是基于这个 文章)

解决方案

部署同步

假设我们正在对前端和后端进行重大更改，并且两者都将与以前的版本不兼容.所以新版本必须同时部署.

在我们当前的设置中，我们必须首先部署后端(这会破坏已部署的前端)，然后部署新的前端，修复生产，但要关闭".期间.

我不是搬运工用户，但也许你可以依靠一些 docker-compose.yml 文件左右，收集后端和前端的版本?在这种情况下，它们可以同时更新……

确实根据 portainer/portainer#1963 和 this doc page，portainer 似乎同时支持 docker-compose 和 swarm 堆栈.

此外，docker swarm 提供了一些功能来执行服务升级而无需停机，如 这个博客，不过不知道这个能在portainer中配置到什么程度.

<块引用>

可能的解决方案

我不确定这里应该使用什么来指定版本:commit hash、git tag、branch、docker image version... 最后一个可能避免重建和测试图像，但我认为图像名称和版本在 Portainer 的堆栈定义中是固定的，不容易自动更新.

虽然提交哈希是精确的标识符，但它们可能不足以识别不兼容的版本.因此，您可能希望在 Git 后端存储库中使用标签(和/或分支)来依赖 语义版本控制.p>

然后，您可以相应地标记相应的 Docker 映像，如果需要，可以引入一些同义标记.例如，假设后端已发布版本 1.0.0, 1.0.1, 1.1.0, 1.1.1, 1.2.0, 1.2.1, 1.2.2，标准做法包括像这样标记 Docker 图像:

• project/backend:2.0.2 = project/backend:2.0 = project/backend:2
• 项目/后端:2.0.1
• 项目/后端:2.0.0
• project/backend:1.1.1 = project/backend:1.1 = project/backend:1
• 项目/后端:1.1.0
• project/backend:1.0.1 = project/backend:1.0
• 项目/后端:1.0.0

(如果需要，删除旧图像)

<块引用>

后端集成测试

目前后端未针对前端进行测试(仅另一种方式).

好的，但我想你的方法是相当标准的(前端取决于后端，而不是相反).

无论如何，我记得即使被测系统是前端，实施单元测试(开发和运行成本低于集成测试)可能是值得的，以便管道的第一阶段在触发必要的集成测试之前快速运行这些单元测试.

<块引用>

测试的分支依赖

在我们当前的设置中，前端的所有提交都针对已部署的后端进行测试(为避免在 CI 中复制后端，仅使用生产 API 地址)，在这种情况下会导致错误的测试结果.

这可能不够灵活:通常，CI/CD 假定集成测试使用专用后端实例(开发"服务器或预生产"服务器)运行，并且如果所有集成测试和系统测试通过，映像被部署到prod"；服务器(和监控等)

我从你的帖子中看到你正在使用 GitLab CI，它有一些 原生 Docker 支持，所以也许这很容易实现.

几个提示:

• 假设后端已在非向后兼容版本中进行了修改，并且相应的 Docker 映像在注册表中可用(例如 GitLab CI 的).然后您可以在前端配置中更改该图像的规范(例如，在 GitLab CI conffile 中将 project/backend:1 替换为 project/backend:2 左右).

• 您的后端可能是作为 REST Web 服务实现的，在这种情况下，您可能还想在 URL 中添加版本前缀，以便在从 project/backend:1 到 project/backend:2(有不兼容的更改)，如果需要，两个版本可以同时部署到 URL https://example.com/api/v1/… 和 https://example.com/api/v2/…

此外，除了只有两个带有 CI/CD 的存储库的解决方案(后端分开测试，前端针对后端的相关版本进行测试)之外，您最初建议的解决方案也可以考虑:

<块引用>

对于部署同步问题，我考虑创建另一个存储库，其中只有一个文件指定应部署的前端和后端版本.此存储库中的提交将导致 Portanier 的两个服务 webhook 都被卷曲".用于更新(后端和前端).这并不能保证同时更新(在 Portainer 中可能会失败并且不会回滚)，但它会比当前设置更好.

您可以稍微修改此方法以避免出现此类部署失败:您可以将一些 CI 设置添加到第三个 repo，它只包含一个 docker-compose.yml 文件左右，然后移动从前端 CI 到组合"的集成测试；CI…

(仅供参考，此方法类似于此 DigitalOcean 教程，通过一些 docker-compose.test.yml 文件实现了集成测试.)

I'll describe my setup to make the problems less abstract, but they don't seem specific to my case.

Context

We have Python-Django backend and a VueJS frontend, each in a repository, with Gitlab-CI configured and deploy using Portainer (using a stack). A commit in the production branch of each repository follows this path:

1. commit
2. gitlab-ci pipeline:
   1. build docker image
   2. test image (frontend is tested against deployed backend)
   3. tag image as production:latest
   4. push image back to gitlab registry
   5. webhook the corresponding service (frontend/backend) in portainer to update the deployed image

3. portainer:
   1. pull image
   2. deploy

Problems

Deployment synchronization

Imagine we're doing a major change in both frontend and backend, and both will become incompatible with previous versions. So the new versions must be deployed simultaneously.

In our current setup we have to first deploy the backend (what will break the deployed frontend) and then deploy the new frontend, fixing production, but with a "down" period.

Branch dependency for tests

Sometimes when we develop branch feature-1 in the frontend, it must be tested against branch feature-1 from the backend.

In our current setup all the commits in the frontend are tested against the deployed backend (to avoid replicating the backend in CI, only the production API address is used), resulting in false tests results in such cases.

Backend integration tests

When a commit is done to the backend, it can break the frontend.

Currently the backend isn't tested against the frontend (only the other way).

Possible Solutions

For the deployment synchronization problem I thought about creating another repository that would have only one file specifying the versions for frontend and backend that should be deployed. A commit in this repository would result in both Portanier' services webhooks being "curled" for update (backend and frontend). This doesn't guarantee the simultaneous update (one may fail in Portainer and there would be no rollback), but it would be better than current setup.

I'm not sure about what should be used to specify versions here: commit hash, git tag, branch, docker image version... The last maybe avoids having to rebuild and test images, but I think images name and versions are fixed in Portainer' stacks definition, and not easy to update automatically.

For the branch dependency tests I thought about having a file in each repository (frontend and backend) specifying which branch from the backend/frontend to test against. But the CI for each repository would have to replicate the whole deploy environment (running a new backend and frontend to test each frontend commit, for example). This would also allow backend integration tests. Since we're using Docker, that's not very complicated, but will take extra time for each CI pipeline... Also, when the first repository (frontend or backend) is committed, it'll reference a still non existent branch in the other repository, and fail...

These solutions seem awkward to me, specially if these are problems common to CI/CD with Docker. And it can get even uglier when we add more repositories to the mix.

Alternatives?

Thanks for the attention!

(edit: for curious, my current setup was based on this article)

解决方案

Deployment synchronization

Imagine we're doing a major change in both frontend and backend, and both will become incompatible with previous versions. So the new versions must be deployed simultaneously.

In our current setup we have to first deploy the backend (what will break the deployed frontend) and then deploy the new frontend, fixing production, but with a "down" period.

I'm not a portainer user, but maybe you could rely on some docker-compose.yml file or so, gathering both the version of the backend and the frontend? in this case they could be updated at the same time…

Indeed according to portainer/portainer#1963 and this doc page, portainer seems to support both docker-compose and swarm stacks.

Also, docker swarm provides some features to perform service upgrade without downtime, as documented in this blog, but I don't know to what extent this can be configured in portainer.

Possible Solutions

I'm not sure about what should be used to specify versions here: commit hash, git tag, branch, docker image version... The last maybe avoids having to rebuild and test images, but I think images name and versions are fixed in Portainer' stacks definition, and not easy to update automatically.

While commit hashes are precise identifiers, they are probably not convenient enough to identify incompatible versions. So you may want to rely on semantic versioning using tags (and/or branches) on your Git backend repo.

Then, you may tag the corresponding Docker images accordingly, introducing some synonymous tags if need be. For example, assuming the backend has been released with versions 1.0.0, 1.0.1, 1.1.0, 1.1.1, 1.2.0, 1.2.1, 1.2.2, a standard practice consists in tagging the Docker images like this:

• project/backend:2.0.2 = project/backend:2.0 = project/backend:2
• project/backend:2.0.1
• project/backend:2.0.0
• project/backend:1.1.1 = project/backend:1.1 = project/backend:1
• project/backend:1.1.0
• project/backend:1.0.1 = project/backend:1.0
• project/backend:1.0.0

(removing old images if need be)

Backend integration tests

Currently the backend isn't tested against the frontend (only the other way).

OK but I guess your approach is fairly standard (the frontend depends on the backend, not the other way around).

Anyway, I recall that even if the system under test is a front-end, it may be worth it to implement unit tests (which are less costly to develop and run than integration tests) so that a first stage in the pipeline quickly runs these unit tests, before triggering the necessary integration tests.

Branch dependency for tests

In our current setup all the commits in the frontend are tested against the deployed backend (to avoid replicating the backend in CI, only the production API address is used), resulting in false tests results in such cases.

This may be not flexible enough: in general, CI/CD assumes the integration tests are run using a dedicated backend instance ("dev" server or "pre-prod" server), and if all integration tests and system tests pass, the image is deployed to the "prod" server (and monitored, etc.)

I see from your post that you are using GitLab CI, which has some native Docker support, so maybe this could be implemented easily.

A couple of hints:

• Assume the backend has been modified in a non-backward compatible version, and the corresponding Docker image is available in a registry (e.g. that of GitLab CI). Then you could just change the specification of that image in the frontend configuration (e.g., replacing project/backend:1 with project/backend:2 or so in the GitLab CI conffile).

• Your backend is probably implemented as a REST Web Service, in which case you might also want to add a version prefix in your URL, so that when you switch from project/backend:1 to project/backend:2 (with incompatible changes), both versions could be deployed at the same time if need be, to the URLs https://example.com/api/v1/… and https://example.com/api/v2/…

Also, beyond the solution to have only two repos with CI/CD (backend tested apart, and frontend tested against the relevant version of the backend), the solution you suggested in the first place may also be considered:

For the deployment synchronization problem I thought about creating another repository that would have only one file specifying the versions for frontend and backend that should be deployed. A commit in this repository would result in both Portanier' services webhooks being "curled" for update (backend and frontend). This doesn't guarantee the simultaneous update (one may fail in Portainer and there would be no rollback), but it would be better than current setup.

You could slightly modify this approach to avoid one such deployment failure: you could add some CI setup to that third repo, that would only contain a docker-compose.yml file or so, and move the integration tests from the frontend CI to that "compose" CI…

(FYI this approach is similar to the one suggested in this DigitalOcean tutorial, where the integration testing is achieved thanks to some docker-compose.test.yml file.)

这篇关于在 CI/CD 中如何管理前端和后端之间的依赖关系?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！