S3 - 究竟什么是前缀?什么速率限制适用? [英] S3 - What Exactly Is A Prefix? And what Ratelimits apply?

问题描述

我想知道是否有人知道 s3 前缀究竟是什么以及它如何与亚马逊的 已发布的 s3 速率限制:

<块引用>

Amazon S3 自动扩展到高请求率.例如，您的应用程序可以实现至少 3,500 PUT/POST/DELETE 和 5,500存储桶中每个前缀的每秒 GET 请求数.没有限制一个桶中的前缀数.

虽然这很清楚，但我不太确定前缀是什么?

前缀是否需要分隔符?

如果我们有一个存储桶，我们将所有文件存储在根"级别(完全平坦，没有任何前缀/分隔符)是否算作单个前缀"并且是否受上面发布的速率限制的约束?

>

我解释亚马逊文档的方式 向我表明情况就是如此，并且扁平结构将被视为单个前缀".(即它会受到上面公布的速率限制)

<块引用>

假设您的存储桶(管理员创建的)有四个对象以下对象键:

开发/项目1.xls

财务/报表1.pdf

私人/税务文件.pdf

s3-dg.pdf

s3-dg.pdf 键没有前缀，所以它的对象出现直接在存储桶的根级别.如果你打开 Development/文件夹，您会在其中看到 Projects.xlsx 对象.

在上面的示例中，s3-dg.pdf 是否会受到与其他每个前缀(Development/Finance/Private)不同的速率限制(5500 GET 请求/秒)?><小时>

更令人困惑的是我读过几篇关于亚马逊使用前 N 个字节作为分区键并鼓励使用高基数前缀的博客，我只是不确定它如何与带有flat"的存储桶交互文件结构".

解决方案

你说得对，公告似乎自相矛盾.它只是写得不正确，但信息是正确的.简而言之:

1. 每个前缀每秒最多可实现 3,500/5,500 个请求，因此对于许多用途，假设是您不需要使用多个前缀.
2. 前缀被认为是对象位置的整个路径(直到最后一个/")，并且不再仅按前 6-8 个字符进行散列.因此，只需在任意两个文件夹"之间拆分数据即可实现每秒 x2 最大请求.(如果请求在两者之间平均分配)

作为参考，这里是 AWS 支持对我的澄清请求的回应:

<块引用>

你好奥伦，

感谢您联系 AWS Support.

我了解您阅读了有关 S3 请求率性能的 AWS 帖子正在增加，您对此还有其他问题公告.

在这次升级之前，S3 支持 100 个 PUT/LIST/DELETE 请求秒和每秒 300 个 GET 请求.为了获得更高的性能，必须实施随机散列/前缀模式.从去年开始请求速率限制增加到 3,500 PUT/POST/DELETE 和 5,500每秒 GET 请求.这种增加通常足以减轻 503 SlowDown 错误的应用程序随机化前缀.

然而，如果新的限制不够，前缀需要使用.前缀没有固定的字符数.它是任何字符串存储桶名称和对象名称之间，例如:

• bucket/folder1/sub1/file
• bucket/folder1/sub2/file
• bucket/1/file
• bucket/2/file

对象文件"的前缀为: /folder1/sub1/ ，/folder1/sub2/、/1/、/2/.在这个例子中，如果你传播读平均跨越所有四个前缀，您可以实现每个请求 22,000 个第二个.

I was wondering if anyone knew what exactly an s3 prefix was and how it interacts with amazon's published s3 rate limits:

Amazon S3 automatically scales to high request rates. For example, your application can achieve at least 3,500 PUT/POST/DELETE and 5,500 GET requests per second per prefix in a bucket. There are no limits to the number of prefixes in a bucket.

While that's really clear I'm not quite certain what a prefix is?

Does a prefix require a delimiter?

If we have a bucket where we store all files at the "root" level (completely flat, without any prefix/delimters) does that count as single "prefix" and is it subject to the rate limits posted above?

The way I'm interpreting amazon's documentation suggests to me that this IS the case, and that the flat structure would be considered a single "prefix". (ie it would be subject to the published rate limits above)

Suppose that your bucket (admin-created) has four objects with the following object keys:

Development/Projects1.xls

Finance/statement1.pdf

Private/taxdocument.pdf

s3-dg.pdf

The s3-dg.pdf key does not have a prefix, so its object appears directly at the root level of the bucket. If you open the Development/ folder, you see the Projects.xlsx object in it.

In the above example would s3-dg.pdf be subject to a different rate limit (5500 GET requests /second) than each of the other prefixes (Development/Finance/Private)?

---

What's more confusing is I've read a couple of blogs about amazon using the first N bytes as a partition key and encouraging about using high cardinality prefixes, I'm just not sure how that interacts with a bucket with a "flat file structure".

解决方案

You're right, the announcement seems to contradict itself. It's just not written properly, but the information is correct. In short:

1. Each prefix can achieve up to 3,500/5,500 requests per second, so for many purposes, the assumption is that you wouldn't need to use several prefixes.
2. Prefixes are considered to be the whole path (up to the last '/') of an object's location, and are no longer hashed only by the first 6-8 characters. Therefore it would be enough to just split the data between any two "folders" to achieve x2 max requests per second. (if requests are divided evenly between the two)

For reference, here is a response from AWS support to my clarification request:

Hello Oren,

Thank you for contacting AWS Support.

I understand that you read AWS post on S3 request rate performance being increased and you have additional questions regarding this announcement.

Before this upgrade, S3 supported 100 PUT/LIST/DELETE requests per second and 300 GET requests per second. To achieve higher performance, a random hash / prefix schema had to be implemented. Since last year the request rate limits increased to 3,500 PUT/POST/DELETE and 5,500 GET requests per second. This increase is often enough for applications to mitigate 503 SlowDown errors without having to randomize prefixes.

However, if the new limits are not sufficient, prefixes would need to be used. A prefix has no fixed number of characters. It is any string between a bucket name and an object name, for example:

• bucket/folder1/sub1/file
• bucket/folder1/sub2/file
• bucket/1/file
• bucket/2/file

Prefixes of the object 'file' would be: /folder1/sub1/ , /folder1/sub2/, /1/, /2/. In this example, if you spread reads across all four prefixes evenly, you can achieve 22,000 requests per second.

这篇关于S3 - 究竟什么是前缀?什么速率限制适用?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！