将AWS SQS用作Aurora数据库的写入队列以提高系统性能是否有效 [英] Is it valid to use AWS SQS as a write queue to Aurora database to increase system performance
问题描述
我正在AWS上开发Web应用程序服务器，该服务器需要在读写时支持高吞吐量.老板给了我这样的高级设计.
I am developing a web application server on AWS that need to support high throughput on reading and write. My boss gave me a high-level design like this.
我被困在写队列"中.团队告诉我，我们需要它来提高写入性能，因为我们只能写入1个主副本.我对消息队列(例如SQS和RabbitMQ)有一些基本的了解，但是对将其用作数据库写队列一无所知.
I am stuck on the "Write Queue". The team told me that we need it to increase the performance of writing because we can have only 1 master replica to which we can write. I have some basic knowledge about message queues such as SQS and RabbitMQ but don't know anything about using it as a database write queue.
目前，我有3个问题:
-
使用这种体系结构，真的能够提高写入数据库的性能(与直接写入主副本相对)吗?
Using this architecture, is it really able to increase the performance of writing to the database (as opposed to writing directly to the master replica).
当在写入过程中发生错误时，如何处理事务，尤其是如何回滚.通常，我们将使用应用程序代码控制事务，以便在发生错误时，整个事务都将回滚，并且App Server会使用一些错误代码来响应客户端.
How to handle transactions, especially how to rollback, when errors occur during the writing. Normally, we would control the transaction in application code such that when an error occurs, the whole transaction is rollback and the App Server response to the client with some error code.
我提到我已经研究过将消息队列用作写队列，但是我不确定是否朝着正确的方向前进.也许，已经有其他一些技术适合作为数据库的写队列了?
I mentioned that I have researched about using message queue as a write queue, but I am not sure if I am looking in the right direction. Maybe, there is some other technology already that is suitable to be a write queue to the database?
除了这些问题之外，我相信这应该是一个大话题，并且想知道我可以在其中进行详细研究的资源.
In addition to the questions, I believe this should be a big topic and would like to know the resources where I can research in detail on this topic.
推荐答案
在类似情况下，使用队列作为解耦两个系统的手段.实现此类体系结构模式时，存在许多优点和缺点.我将尝试列出我认为是主要的内容.
In similar cases, queues are used as a mean for de-coupling two systems. There are several advantages and disadvantages when implementing such architectural patterns. I will try to list what I believe are the main ones.
优势
-
缩短了响应时间
Improved response time
由于队列不需要复杂的事务，因此它们通常是一种快速且安全(如果配置正确)的存储.这意味着来自客户端的感知响应延迟将减少，从而给人一种服务更快"的感觉.
As queues do not require complex transactions they are usually a fast and, if correctly configured, safe storage. This means the perceived response latency from the client side will decrease giving the feeling that the service is "faster".
关注点分离
正确地解耦服务可以提高其对错误的恢复能力.例如，如果数据库不能接受更多的写请求，则客户端将不受影响，并且它们的请求也不会丢失，因为它们将位于队列中.这使运营商有更多时间对问题做出反应，而服务价值仅受到部分影响.
Correctly de-coupling services increases their resilience to errors. For example, if the DB cannot accept more write requests, the clients will be unaffected and their requests will still not be lost as they will be in the queue. This gives Operators more time to react to problems while the service value is only partially affected.
可扩展性
当操作变得复杂时，通常最好将它们分成微型组件.与单片服务相比，放大微组件要容易得多.作业队列启用了这种设计模式.
When operations become complex, it's usually a good idea to separate them into microcomponents. It is way easier to scale up microcomponents than monolithic services. Job queues enable such design patterns.
缺点
-
从错误中恢复变得更加复杂
Recovering from errors becomes more complex
如上所述，如果DB停止接受请求，则作业将堆积在队列中.现在您有两个问题要处理:完整的数据库和完整的作业队列.系统问题开始在整个体系结构中传播，例如涟漪会引起多种副作用，并使人们难以理解根本原因.
As said above, if the DB stops accepting requests, jobs will pile up in the queue. Now you have 2 problems to deal with: a full DB and a full job queue. System problems start propagating across your architecture like ripples causing several side effects and making hard to understand what is the root cause.
识别瓶颈需要更多时间
如果数据库写入速度很慢，则在其前面放一个队列不会使事情变快.乔布斯仍将排在队列中，您的下一个任务将是弄清楚为什么会发生这种情况.在处理复杂的ETL管道时，提高性能成为一项繁琐的操作，您的瓶颈将在系统之间转移.
If the DB writes are slow, putting a queue in front of it won't make things faster. Jobs will still pile up in the queue and your next task will be figuring out why this happens. When dealing with complex ETL pipelines, improving performance becomes a quite tedious whack-a-mole operation where your bottlenecks just shift from system to system.
每次操作的费用增加
一项工作完成所需经过的阶段越多，该工作将需要更多的时间和金钱.
The more stages a job needs to traverse for its completion, the more time and money that job will require.
去耦组件通常被视为处理性能问题的灵丹妙药.正确分离关注点和责任是非常有益的做法，但需要大量的经验和护理.如今，单一服务已被视为一切邪恶的根源.就我个人而言，我更喜欢处理一大堆意大利面条而不是分散的意大利面条.
De-coupling components is usually seen as a silver bullet for dealing with performance issues. The correct separation of concerns and responsibilities is a very beneficial practice but requires a lot of experience and care. Nowadays monolithic services are seeing as the root of all evils. Personally I prefer to deal with a monolithic bunch of spaghetti rather than a distributed one.
这篇关于将AWS SQS用作Aurora数据库的写入队列以提高系统性能是否有效的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
