亚马逊网络服务 S3 请求限制 [英] Amazon Web Services S3 Request Limit

问题描述

我正在使用 AWS 运行一些数据处理.我在 EC2 中有 400 个 Spot 实例，每个实例有 4 个进程，它们都写入 S3 中的单个存储桶.我开始收到一个(显然不常见的)错误说:

I'm using AWS to run some data processing. I have 400 spot instances in EC2 with 4 processes each, all of them writing to a single bucket in S3. I've started to get a (apparently uncommon) error saying:

503:慢下来

有谁知道 S3 存储桶的实际请求限制是多少?我在上面找不到任何 AWS 文档.

Does anyone know what the actual request limit is for an S3 bucket? I cannot find any AWS documentation on it.

谢谢！

推荐答案

据我所知，Slow Down 是一个非常罕见的错误.但是，在发布此问题后，我收到了来自 AWS 的一封电子邮件，其中说我的 LIST 请求已限制为每秒 10 个请求，因为我有太多请求进入特定存储桶.

From what I've read, Slow Down is a very infrequent error. However, after posting this question I received an email from AWS that said the had capped my LIST requests to 10 requests per second because I had too many going to a specific bucket.

我一直在为我正在处理的项目使用自定义排队脚本，该脚本依赖 LIST 请求来确定要处理的下一个项目.遇到这个问题后，我改用 AWS SQS，这比我想象的要简单得多.不再有自定义队列，不再有大量的 LIST 请求.

I had been using a custom queuing script for the project I am working on, which relied on LIST requests to determine the next item to process. After running into this problem I switched to AWS SQS, which was a lot simpler to implement than I'd thought it would be. No more custom queue, no more massive amount of LIST requests.

感谢您的回答！

这篇关于亚马逊网络服务 S3 请求限制的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！