如何解决 OpenShift 集群中多个 pod 中的 liquibase 等待更改日志锁定问题? [英] How to solve liquibase waiting for changelog lock problem in several pods in OpenShift cluster?
问题描述
我们支持使用 Spring Boot 用 Java 编写并部署在 OpenShift 中的多个微服务.一些微服务与数据库通信.我们经常在单个部署中的多个 Pod 中运行单个微服务.当每个微服务启动时，它会启动 liquibase，它会尝试更新数据库.问题是有时一个 pod 在等待更改日志锁定时会失败.当这种情况发生在我们的生产 OpenShift 集群中时，我们预计其他 pod 在重新启动时会失败，因为与更改日志锁定问题相同.因此，在最坏的情况下，所有 Pod 都将等待解除锁定.
We are supporting several microservices written in Java using Spring Boot and deployed in OpenShift. Some microservices communicate with databases. We often run a single microservice in multiple pods in a single deployment. When each microservice starts, it starts liquibase, which tries to update the database. The problem is that sometimes one pod fails while waiting for the changelog lock. When this happens in our production OpenShift cluster, we expect other pods to fail while restarting because of the same problem with changelog lock issue. So, in the worst case scenario, all pods will wait for the lock to be lifted.
我们希望 Liquidbase 在每个 Pod 启动时自动准备我们的数据库架构.
We want liquidbase to automatically prepare our database schemas when each pod is starting.
在每个微服务中都存储这个逻辑好吗?出现liquidbase changelog lock问题如何自动解决?我们需要将数据库准备逻辑放在单独的部署中吗?
Is it good to store this logic in every microservice? How can we automatically solve the problem when the liquidbase changelog lock problem appears? Do we need to put the database preparation logic in a separate deployment?
所以也许我应该解释一下我的问题.就微服务架构而言，运行数据库迁移的最佳方式是什么?也许我们不应该在每个 pod 中使用数据库迁移?也许最好用单独的部署来完成，或者用一些根本不在 OpenShift 中的额外 Jenkins 工作来完成?
So maybe I should paraphrase my question. What is the best way to run db migration in term of microservice architecture? Maybe we should not use db migration in each pod? Maybe it is better to do it with separate deployment or do it with some extra Jenkins job not in OpenShift at all?
推荐答案
我们将 liquibase 迁移作为 Kubernetes 中的 init-container 运行.在微服务中运行 Liquibase 的问题是，如果在配置的超时之前就绪探测没有成功，Kubernetes 将终止 pod.在我们的例子中，这有时发生在大型数据库迁移期间，这可能需要几分钟才能完成.Kubernetes 将终止 pod，使 DATABASECHANGELOGLOCK 处于锁定状态.使用 init-containers 你不会有这个问题.请参阅 https://www.liquibase.org/blog/using-liquibase-in-kubernetes 的详细解释.
We're running liquibase migrations as an init-container in Kubernetes. The problem with running Liquibase in micro-services is that Kubernetes will terminate the pod if the readiness probe is not successful before the configured timeout. In our case this happened sometimes during large DB migrations, which could take a few minutes to complete. Kubernetes will terminate the pod, leaving DATABASECHANGELOGLOCK in a locked state. With init-containers you will not have this problem. See https://www.liquibase.org/blog/using-liquibase-in-kubernetes for a detailed explanation.
更新请看一下这个 Liquibase 扩展，它使用数据库锁替换了 StandardLockService:https://github.com/blagerweij/liquibase-sessionlock
UPDATE Please take a look at this Liquibase extension, which replaces the StandardLockService, by using database locks: https://github.com/blagerweij/liquibase-sessionlock
此扩展使用 MySQL 或 Postgres 用户锁定语句，这些语句在数据库连接关闭时(例如，当容器意外停止时)会自动释放.使用扩展所需的唯一事情是向库添加依赖项.Liquibase 会自动检测改进后的 LockService.
This extension uses MySQL or Postgres user lock statements, which are automatically released when the database connection is closed (e.g. when the container is stopped unexpectedly). The only thing required to use the extension is to add a dependency to the library. Liquibase will automatically detect the improved LockService.
我不是该库的作者，但我在寻找解决方案时偶然发现了该库.我通过将库发布到 Maven 中心来帮助作者.目前支持 MySQL 和 PostgreSQL，但应该很容易支持其他 RDBMS.
I'm not the author of the library, but I stumbled upon the library when I was searching for a solution. I helped the author by releasing the library to Maven central. Currently supports MySQL and PostgreSQL, but should be fairly easy to support other RDBMS.
这篇关于如何解决 OpenShift 集群中多个 pod 中的 liquibase 等待更改日志锁定问题?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
