Google数据存储 - 当您超过每秒一次写入限制时会发生什么？ [英] Google Datastore - What happens when you exceed the one write per second limit?

问题描述

我试图创建约100,000个具有相同父代的新实体（代表用户）。我读过每个实体组每秒写入一个实体的限制。我认为这个请求可能会超时，所以我决定使用推送队列任务来延长我需要十分钟的时间。我尝试在推送队列任务的for循环中使用put（），但最终仍然超时（只能编写大约8,900个实体）。

我很困惑，为什么我没有得到错误，因为我试图对同一组进行多次写入。任务超时10分钟，这意味着我每分钟写入890个写入，每秒写入约14个写入。这是每秒写一次的方式..我阅读了答案 Google App Engine HRD - 如果我超过写入实体组的每秒1写限制？和Google数据存储 - 没有看到1每个实体组的限制写入次数，但据我的理解，它只是说数据存储可能每秒写入5-10个实体。我得到的收入高于此收入。

我还阅读了 here

当单个实体或实体组更新过快时，会发生数据存储争用。数据存储将排队并发请求以等待轮到。请求在队列中等待超时时间将抛出并发异常

这是否意味着不会抛出错误超过1写/秒？写入操作只会被放入一个队列中，并且在发生请求超时时（在这种情况下，任务队列为10分钟），我只会收到一个错误？

解决方案

1. 对具有自动缩放的实例运行的任务有10分钟的限制。您可以将100,000个用户分成更小的批次，并在每个单独的任务中处理每个批次。




2. 您可以对数据存储使用批量调用，最多可以保存500个实体一个单一的调用，比单独保存每个实体要快得多。




3. 绝对没有理由让所有用户都在同一个实体组中。这个数据模型会带来负面的性能影响 - 写入限制是有原因的。实体组被设计为具有3个地址或10个相册的用户，即使这样，我几乎总是避免亲子关系，因为他们很少添加任何值，但使代码更复杂（您始终必须知道父代检索或保存一个实体）。

I'm trying to create about 100,000 new entities (representing users) that have the same parent. I read that there is a limit of one entity write per second per entity group. I thought the request may time out so I decided to use a Push Queue Task to extend the time I had to ten minutes. I tried using put() in a for loop in a Push Queue Task, but I ended up timing out still (only got to write about 8,900 entities).

I'm confused as to why I didn't get an error since I tried to do multiple writes to the same group. The task timed out at 10 minutes so that means I got 890 writes per minute, which is about 14 writes per second. This is way over one write per second.. I read the answers for Google App Engine HRD - what if I exceed the 1 write per second limit for writing to the entity group? and Google Datastore - Not Seeing 1 Write per Second per Entity Group Limitation, but to my understanding it just says that it's possible for the Datastore to write 5-10 entities per second. The rate I got was higher than that though.

I also read here that

Datastore contention occurs when a single entity or entity group is updated too rapidly. The datastore will queue concurrent requests to wait their turn. Requests waiting in the queue past the timeout period will throw a concurrency exception

Does this mean that an error won't be thrown for trying to exceed 1 write/sec? The writes will just get placed in a queue and I'll only get an error when the timeout for the request occurs (in this case 10 minutes for the Task Queue)?

解决方案

1. You are bumping into 10 minute limitation on tasks that run on instances with automatic scaling. You can split your 100,000 users into smaller batches and process each batch in a separate task.

2. You can use batch calls to the datastore saving up to 500 entities in a single call, which is much faster than saving each entity individually.

3. There are absolutely no reasons to have all users in the same entity group. This data model will have negative performance implications - the write limit is there for a reason. Entity groups are designed for something like a user with 3 addresses or 10 photo albums, and even then I almost always avoid parent-child relationships as they rarely add any value, but make the code more complex (you always have to know the parent to retrieve or save an entity).

这篇关于Google数据存储 - 当您超过每秒一次写入限制时会发生什么？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！