DynamoDB 配置的写入容量单位经常意外地超出 [英] DynamoDB provisioned Write Capacity Units exceeded too often and unexpectedly

问题描述

我相信我了解写入/读取容量单位、它们的工作方式以及在 DynamoDB 中的计算方式.证明是我理解 .

这些项目肯定小于 1KB.它们看起来像这样:

<代码>{"reading_id": "<当前时间>","sensor_id": "","湿度": "<湿度>","温度": "<温度>"}

因为在文档中，它说:

<块引用>

time.sleep(secs)

在给定的秒数内暂停当前线程的执行.参数可能是一个浮点数，以指示更精确的睡眠时间.实际的暂停时间可能比请求的要少，因为任何捕获的信号都会在执行该信号的捕获例程后终止 sleep().此外，由于系统中其他活动的调度，暂停时间可能比任意数量的请求长.

我对 python 不是很熟悉，这让我觉得它可能是我的代码中的东西.这并不能解释我在今年早些时候没有遇到这个问题的事实.

任何人知道上述问题的答案，或者我应该在哪里进一步调查这个问题?

注意:我在这里搜索了 Google 和其他相关问题.似乎没有一个适用于我的情况.

谢谢.

解决方案

您正在共享的图表显示的是一分钟内聚合的消耗量.也就是说，对于图表上的每个数据点，每 60 秒消耗的所有容量的总和.

当您配置一个具有 5 个 WCU 的表时，这意味着您每秒最多只能写入 5 个 1KB 的项目.这实际上为您提供了每分钟最多可使用的 300 个 WCU.

所以，只要你看到 6 个左右的数据点就完全没问题.

需要注意的一点是，预置写入吞吐量的总和(橙色线)实际上不是总和.这似乎是 CloudWatch 中的一个错误:它是每秒预配置的吞吐量.

一个小观察:您每分钟显示 5-6 个单位，这意味着您实际上睡眠时间接近 10 秒，而不是写入之间的 5 秒.

最后，使用 Dynamo，您需要为预留的容量付费，而不是为消耗的容量付费.因此，只要您的表没有受到限制，即使您略微超出了预配置的容量(Dynamo 在某些情况下允许)，您也不会被收取额外费用.

I believe I understand Write/Read capacity units, how they work and are calculated in DynamoDB. Proof of that is that I understand this article thoroughly as well as the aws documentation. That said I'm experiencing an unexpected behavior when writing items to my table.

I have a DynamoDB table with the following settings. Most notably 5 Write/Read Capacity Units

I'm putting in this table readings from sensors connected to a Raspberry Pi that I get and send with python2.7 to Dynamo with my script.

This items are less than 1KB for sure. They look like this:

{
    "reading_id": "<current_time>",
    "sensor_id": "<SENSORS_IDS[i]>",
    "humidity": "<humidity>",
    "temperature": "<temperature>"
}

My script iterates over the sensors, reads from one and submits the sensor's reading/item to DynamoDB with table.put_item every 5 seconds. That is, if the read from the sensors was successful otherwise arbitrarily wait 30s.

Now according to my calculations, I'm writing to DynamoDB 1KB item every 5 seconds, which should be fine since my table is setup up with 5WCU = (5items*1KB)/Second write throughput.

So my questions are:

1. How is it that this small load (if happens as I believe is happening) is exceeding my 5 WCU as seen here?:

2. I have been operating with this setup without changes for about a year(free tier ends September 30, 2018). How is it that, it began to change a few months ago even before the free tier ended, as seen here?:

My only suspect so far is time.sleep() since in the documentation, it says:

time.sleep(secs)

Suspend execution of the current thread for the given number of seconds. The argument may be a floating point number to indicate a more precise sleep time. The actual suspension time may be less than that requested because any caught signal will terminate the sleep() following execution of that signal’s catching routine. Also, the suspension time may be longer than requested by an arbitrary amount because of the scheduling of other activity in the system.

I am not very familiar with python which makes me think that it could be something in my code. That doesn't explain the fact that I was not having this issue earlier in the year though.

Anyone has any idea of the answers to the questions above or where should I investigate this issue further?

Note: I searched Google and other related questions here. None seemed to apply to my case.

Thank you.

解决方案

The graph you are sharing is showing consumption aggregated over a minute. That is, the sum of all capacity consumed over each 60 seconds, for each data point on the graph.

When you provision a table with 5 WCU then that means you can only write up to 5 1KB items each second. Effectively that gives you a total of up to 300 WCU that you could use in each minute.

So, as long as you see data points of 6 or so then that is totally fine.

One thing to notice is the sum of provisioned write throughput (the orange line) is actually not a sum. That seems to be a bug in CloudWatch: it is instead the per-second provisioned throughput.

A minor observation: you are showing 5-6 units each minute, that means you are actually sleeping for closer to 10 seconds, not 5 between writes.

Lastly, with Dynamo you pay for the capacity that you reserve, not what you consume. So as long as your table is not being throttled, even when you go slightly over the provisioned capacity (which Dynamo allows in certain cases) you will not be charged extra.

这篇关于DynamoDB 配置的写入容量单位经常意外地超出的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！