估计SQS处理时间和负载 [英] Estimate SQS processing time and load

问题描述

我将使用AWS SQS(常规队列，而不是FIFO)来处理不同的客户端指标.

I am going to use AWS SQS(regular queue, not FIFO) to process different client side metrics.

我希望每秒大约有400条消息(最坏的情况).我的SQS消息将包含文件的S3位置.

I’m expect to have ~400 messages per second (worst case).My SQS message will contain S3 location of the file.

我创建了一个应用程序，它将监听我的SQS队列，并处理来自它的消息.

I created an application, which will listen to my SQS Queue, and process messages from it.

按流程，我的意思是:

• 阅读SQS消息->
• 从该SQS消息中获取S3位置->
• 呼叫S3客户端->
• 读取该文件->
• 添加一些其他字段->
• 将此文件中的数据发布到AWS Kinesis Firehose.

将对队列中的每个SQS消息进行类似的处理. S3文件的大小很小，小于0.5 KB.

Similar process will be for each SQS message in the Queue. The size of S3 file is small, less than 0,5 KB.


如何计算我是否每秒能够处理这400条消息?如何估计我的解决方案可以处理5倍的数据增长?

How can calculate if I will be able to process those 400 messages per second? How can I estimate that my solution would handle x5 increase in data?

推荐答案

如何计算我是否每秒能够处理这400条消息?如何估计我的解决方案可以处理5倍的数据增长?

How can calculate if I will be able to process those 400 messages per second? How can I estimate that my solution would handle x5 increase in data?

测试它！ 从小范围开始，然后进行数学运算以从中推断.使您的测试环境尽可能接近生产环境.

Test it! Start with a small scale, and do the math to extrapolate from there. Make your test environment as close to what it will be in production as feasible.

• 在单个主机和单个线程上，数学很简单:
  • 1000 / AvgTotalTimeMillis = AvgMessagesPerSecond或
  • 1000 / AvgMessagesPerSecond = AvgTotalTimeMillis
  • On a single host and single thread, the math is simple:
    • 1000 / AvgTotalTimeMillis = AvgMessagesPerSecond, or
    • 1000 / AvgMessagesPerSecond = AvgTotalTimeMillis

    如何进行测试:

    • 从单个线程和主机开始，并为您概述的每个步骤生成一些计时指标以及总时间.

    • Start with a single thread and host, and generate some timing metrics for each step that you outlined, along with a total time.

    • 弄清楚您的平均/最大/最小时间，以及每秒转化为多少条消息
    在单个线程上每秒
    • 400条消息&主机将在每条消息3ms之内.希望这很明显，您需要多个线程/主机.
    • Figure out your average/max/min time, and how many messages per second that translates to
    • 400 messages per second on a single thread & host would be under 3ms per message. Hopefully this makes it obvious you need multiple threads/hosts.

    向上扩展！

    • 现在您知道一个线程可以处理多少个线程，找出一个主机可以有效处理多少个线程(您需要进行实验).在可能的情况下考虑批处理消息-SQS提供批处理操作.
    • 使用数学计算所需的主机数量
    • 如果您需要该数字的5倍，请从那里上移

    在进行此数学运算时，请考虑所用系统的任何限制:

    While you're doing this math, consider any limits of the systems you're using:

    • 查看SQS/S3/Firehose等的节流限制.如果计划使用Lambda代替EC2进行工作，它也有限制.确保您在这些限制之内，如果您即将超出这些限制，请考虑与AWS支持联系.

    基于我的经验的其他一些建议:

    A few other suggestions based on my experience:

    • 基于您的工作流程概述&详细信息，使用EC2，您可能可以为每个主机处理相当数量的线程
    • M5.large 应该足够了-您可能会变小，因为性能瓶颈很可能是网络I/O来获取和发送消息.
    • 在需要增加吞吐量的情况下，考虑使用自动缩放来处理消息尖峰，尽管请记住，自动缩放可能需要花费几分钟的时间.
    • Based on your workflow outline & details, using EC2 you can probably handle a decent number of threads per host
    • M5.large should be more than enough - you can probably go smaller, as the performance bottleneck will likely be networking I/O to fetch and send messages.
    • Consider using autoscaling to handle message spikes for when you need to increase throughput, though keep in mind autoscaling can take several minutes to kick in.

    这篇关于估计SQS处理时间和负载的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！