如何解决OpenShift集群中多个Pod中的liquibase等待更改日志锁定问题? [英] How to solve liquibase waiting for changelog lock problem in several pods in OpenShift cluster?

问题描述

请对这个问题说几句

我们支持使用Spring Boot用Java编写并部署在OpenShift中的多种微服务.一些微服务与数据库通信.我们通常在单个部署中的多个Pod中运行单个微服务.每个微服务启动时，它都会启动liquibase，后者会尝试更新数据库.问题在于，有时一个pod会因等待更改日志锁定问题而失败. 当由于某种原因在生产OpenShift集群中发生这种情况时，由于与变更日志锁定问题相同的问题，我们预计其他Pod在重新启动时会失败.因此，在最坏的情况下，所有吊舱都将等待锁被解除.

We are supporting several microservices written in Java using Spring Boot and deployed in OpenShift. Some microservices communacates with databases. We often run a single microservice in multiple pods in a single deployment. When each microservice starts, it starts liquibase, which tries to update the database. The problem is that sometimes one pod fails with waiting for the changelog lock issue. When for some reason this happens in production OpenShift cluster, we expect other pods to fail while restarting because of the same problem with changelog lock issue. So in the worst case scenario, all pods will wait for the lock to be lifted.

我们希望Liquidbase在pod启动时自动准备数据库模式.

We want liquidbase automatically prepare our database schemas when pod is starting.

因此，请说出解决此问题的最佳方法? 将此逻辑存储在每个微服务中是否很好? 当液基变更日志锁定问题出现时，我们如何自动解决该问题? 我们是否需要将数据库准备逻辑放在单独的部署中?

So please tell the best way to solve this problem? Is it good to store this logic in every microservice? How can we automatically solve the problem when the liquidbase changelog lock problem appears? Do we need to put the database preparation logic in a separate deployment?

谢谢

#Updated

所以也许我应该解释我的问题.从微服务架构的角度来看，运行数据库迁移的最佳方法是什么?也许我们不应该在每个Pod中使用数据库迁移?也许最好单独进行部署，或者完全不在OpenShift中进行一些额外的Jenkins作业?

So maybe I should paraphrase my question. What is the best way to run db migration in term of microservice architecture? Maybe we should not use db migration in each pod? Maybe it is better to do it with separate deployment or do it with some extra Jenkins job not in OpenShift at all?

推荐答案

我们在Kubernetes中将liquibase迁移作为init容器运行.在微服务中运行Liquibase的问题在于，如果在配置的超时之前未准备就绪探测失败，则Kubernetes将终止Pod.在我们的案例中，有时在大型数据库迁移期间会发生这种情况，这可能需要几分钟才能完成. Kubernetes将终止Pod，使DATABASECHANGELOGLOCK处于锁定状态.使用初始化容器，您将不会遇到此问题.参见 https://www.liquibase.org/blog/using-liquibase-in -kubernetes 进行详细说明.

We're running liquibase migrations as an init-container in Kubernetes. The problem with running Liquibase in micro-services is that Kubernetes will terminate the pod if the readiness probe is not successful before the configured timeout. In our case this happened sometimes during large DB migrations, which could take a few minutes to complete. Kubernetes will terminate the pod, leaving DATABASECHANGELOGLOCK in a locked state. With init-containers you will not have this problem. See https://www.liquibase.org/blog/using-liquibase-in-kubernetes for a detailed explanation.

更新 请使用数据库锁查看此Liquibase扩展，它替代了StandardLockService: https://github .com/blagerweij/liquibase-sessionlock

UPDATE Please take a look at this Liquibase extension, which replaces the StandardLockService, by using database locks: https://github.com/blagerweij/liquibase-sessionlock

此扩展程序使用MySQL或Postgres用户锁语句，当数据库连接关闭时(例如，容器意外停止时)，这些语句将自动释放.使用扩展名的唯一要求是将依赖项添加到库中. Liquibase将自动检测改进的LockService.

This extension uses MySQL or Postgres user lock statements, which are automatically released when the database connection is closed (e.g. when the container is stopped unexpectedly). The only thing required to use the extension is to add a dependency to the library. Liquibase will automatically detect the improved LockService.

我不是图书馆的作者，但是在寻找解决方案时偶然发现了图书馆.我通过将库发布到Maven Central帮助作者.当前支持MySQL和PostgreSQL，但是应该很容易支持其他RDBMS.

I'm not the author of the library, but I stumbled upon the library when I was searching for a solution. I helped the author by releasing the library to Maven central. Currently supports MySQL and PostgreSQL, but should be fairly easy to support other RDBMS.

这篇关于如何解决OpenShift集群中多个Pod中的liquibase等待更改日志锁定问题?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！