亚马逊网络服务S3请求限制 [英] Amazon Web Services S3 Request Limit

问题描述

我使用AWS运行一些数据处理。我在EC2 400点情况下，每4个过程，所有的人都写在S3中单个桶。我已经开始得到一个（显然不常见）错误说：

I'm using AWS to run some data processing. I have 400 spot instances in EC2 with 4 processes each, all of them writing to a single bucket in S3. I've started to get a (apparently uncommon) error saying:

503：慢下来

有谁知道实际的请求限制是什么一个S3桶？我找不到它的任何AWS文档。

Does anyone know what the actual request limit is for an S3 bucket? I cannot find any AWS documentation on it.

感谢您！

推荐答案

从我读过，减速是一个非常罕见的错误。然而，张贴了这个问题后，我收到了来自AWS一封电子邮件，说已经封顶我LIST请求每秒10个请求，因为我有太多的去一个特定的桶中。

From what I've read, Slow Down is a very infrequent error. However, after posting this question I received an email from AWS that said the had capped my LIST requests to 10 requests per second because I had too many going to a specific bucket.

我使用的项目我的工作，这依靠的请求列表来确定下一个项目来处理定制队列脚本已。运行到这个问题后，我切换到AWS SQS，这是一个简单得多实施比我想象的要。没有更多的自定义的队列中，列出请求没有更多的巨量。

I had been using a custom queuing script for the project I am working on, which relied on LIST requests to determine the next item to process. After running into this problem I switched to AWS SQS, which was a lot simpler to implement than I'd thought it would be. No more custom queue, no more massive amount of LIST requests.

谢谢你的答案！

这篇关于亚马逊网络服务S3请求限制的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！