将 AWS SQS 用作 Aurora 数据库的写入队列以提高系统性能是否有效 [英] Is it valid to use AWS SQS as a write queue to Aurora database to increase system performance
问题描述
我正在 AWS 上开发需要支持高读写吞吐量的 Web 应用程序服务器.我的老板给了我这样的高级设计.
I am developing a web application server on AWS that need to support high throughput on reading and write. My boss gave me a high-level design like this.
我被困在写入队列"中.团队告诉我，我们需要它来提高写入性能，因为我们只能有 1 个可以写入的主副本.我对 SQS 和 RabbitMQ 等消息队列有一些基本的了解，但对将其用作数据库写入队列一无所知.
I am stuck on the "Write Queue". The team told me that we need it to increase the performance of writing because we can have only 1 master replica to which we can write. I have some basic knowledge about message queues such as SQS and RabbitMQ but don't know anything about using it as a database write queue.
现阶段，我有3个问题:
使用这种架构，是否真的能够提高写入数据库的性能(而不是直接写入主副本).
Using this architecture, is it really able to increase the performance of writing to the database (as opposed to writing directly to the master replica).
在写入过程中发生错误时如何处理事务，尤其是如何回滚.通常，我们会在应用程序代码中控制事务，这样当发生错误时，整个事务会回滚，并且 App Server 以一些错误代码响应客户端.
How to handle transactions, especially how to rollback, when errors occur during the writing. Normally, we would control the transaction in application code such that when an error occurs, the whole transaction is rollback and the App Server response to the client with some error code.
我提到我已经研究过将消息队列用作写入队列，但我不确定我是否在寻找正确的方向.也许，已经有其他一些技术适合作为数据库的写入队列?
I mentioned that I have researched about using message queue as a write queue, but I am not sure if I am looking in the right direction. Maybe, there is some other technology already that is suitable to be a write queue to the database?
除了问题，我相信这应该是一个很大的话题，想知道我可以详细研究这个话题的资源.
In addition to the questions, I believe this should be a big topic and would like to know the resources where I can research in detail on this topic.
推荐答案
在类似的情况下，队列被用作分离两个系统的手段.实现这样的架构模式有几个优点和缺点.我将尝试列出我认为主要的内容.
In similar cases, queues are used as a mean for de-coupling two systems. There are several advantages and disadvantages when implementing such architectural patterns. I will try to list what I believe are the main ones.
优势
缩短响应时间
Improved response time
由于队列不需要复杂的事务，它们通常是一种快速且如果配置正确的安全存储.这意味着来自客户端的感知响应延迟将减少，给人一种服务更快"的感觉.
As queues do not require complex transactions they are usually a fast and, if correctly configured, safe storage. This means the perceived response latency from the client side will decrease giving the feeling that the service is "faster".
关注点分离
正确解耦服务可以提高它们对错误的恢复能力.例如，如果数据库不能接受更多的写请求，客户端将不受影响，他们的请求仍然不会丢失，因为它们将在队列中.这让运营商有更多时间对问题做出反应，而服务价值只会受到部分影响.
Correctly de-coupling services increases their resilience to errors. For example, if the DB cannot accept more write requests, the clients will be unaffected and their requests will still not be lost as they will be in the queue. This gives Operators more time to react to problems while the service value is only partially affected.
改进的可扩展性
当操作变得复杂时，将它们分成微组件通常是个好主意.与单体服务相比，扩展微组件要容易得多.作业队列支持这种设计模式.
When operations become complex, it's usually a good idea to separate them into microcomponents. It is way easier to scale up microcomponents than monolithic services. Job queues enable such design patterns.
缺点
从错误中恢复变得更加复杂
Recovering from errors becomes more complex
如上所述，如果数据库停止接受请求，作业将堆积在队列中.现在你有两个问题需要处理:一个完整的数据库和一个完整的作业队列.系统问题开始在您的架构中传播，如涟漪，会导致多种副作用，并且难以理解根本原因是什么.
As said above, if the DB stops accepting requests, jobs will pile up in the queue. Now you have 2 problems to deal with: a full DB and a full job queue. System problems start propagating across your architecture like ripples causing several side effects and making hard to understand what is the root cause.
识别瓶颈需要更多时间
如果数据库写入速度很慢，在它前面放置一个队列不会使速度更快.工作仍然会堆积在队列中，您的下一个任务将是弄清楚为什么会发生这种情况.在处理复杂的 ETL 管道时，提高性能成为一项非常乏味的操作，您的瓶颈只是从一个系统转移到另一个系统.
If the DB writes are slow, putting a queue in front of it won't make things faster. Jobs will still pile up in the queue and your next task will be figuring out why this happens. When dealing with complex ETL pipelines, improving performance becomes a quite tedious whack-a-mole operation where your bottlenecks just shift from system to system.
每次操作的成本增加
完成一项工作需要经过的阶段越多，该工作所需的时间和金钱就越多.
The more stages a job needs to traverse for its completion, the more time and money that job will require.
解耦组件通常被视为处理性能问题的灵丹妙药.正确分离关注点和责任是一种非常有益的做法，但需要大量的经验和细心.如今，单体服务被视为万恶之源.就我个人而言，我更喜欢处理一大堆意大利面条，而不是分布式意大利面条.
De-coupling components is usually seen as a silver bullet for dealing with performance issues. The correct separation of concerns and responsibilities is a very beneficial practice but requires a lot of experience and care. Nowadays monolithic services are seeing as the root of all evils. Personally I prefer to deal with a monolithic bunch of spaghetti rather than a distributed one.
这篇关于将 AWS SQS 用作 Aurora 数据库的写入队列以提高系统性能是否有效的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
