GAE交易失败和幂等性 [英] GAE transaction failure and idempotency
问题描述
Google App Engine文档包含以下段落：
注意：如果您的应用程序在提交
事务时收到异常，并不总是意味着交易失败。您的
可能会收到DatastoreTimeoutException，
ConcurrentModificationException或DatastoreFailureException
例外情况，在这些情况下事务已被提交并且
最终将成功应用。在可能的情况下，使您的
Datastore交易幂等性，这样如果您重复交易，
，最终结果将是相同的。
等等，什么？看起来有一类非常重要的事务，它们根本不可能变成幂等因为它们依赖于当前的数据存储状态。例如，一个简单的计数器，就像一个像按钮一样。交易需要读取当前计数，增加它，并再次写出计数。如果事务看起来失败，但并不真正失败，并且我无法在客户端告诉它，那么我需要再试一次，这将导致一次点击产生两个喜欢。肯定有一些方法可以用GAE来防止这种情况发生？
编辑：
看起来这是固有的问题在分布式系统中，除了Guido van Rossum以外 - 请参阅此链接：
所以看起来设计幂等事务几乎是必须的，如果你想高度的可靠性。
我想知道是否有可能在整个应用中实施一个全球系统来确保幂等性。关键将是维护数据存储中的事务日志。客户端会生成一个GUID，然后在请求中包含该GUID（对于同一个请求，重试时将重新发送相同的GUID）。在服务器上，在每个事务开始时，它将在数据存储中查找具有该ID的Transactions实体组中的记录。如果它找到了，那么这是一个重复的事务，所以它会在没有做任何事情的情况下返回。
当然，这需要启用跨组交易，或者将单独的事务日志作为每个实体组的子项。如果失败的实体键查找速度很慢，也会有性能问题，因为几乎每个事务都会包含失败的查找，因为大多数GUID都是新的。
额外的数据存储相互作用的额外成本，如果我不得不使每个事务都是幂等的，这可能仍然会少一些，因为这将需要大量的检查每个级别的数据存储中的内容。
dan wilkerson，simon goldsmith，et al。在应用引擎的本地（每个实体组）事务之上设计了一个彻底的全球事务系统。在高层次上，它使用的技术类似于你描述的GUID。丹处理了潜艇写道，即你描述的那些报告失败但后来表面成功的交易，以及数据存储的许多其他理论和实践细节。 erick armbrust在木薯醇中实现了dan的设计。
我并不一定建议您实施他的设计或使用木薯醇，但是您肯定对研究感兴趣。
回答您的问题：许多人实施GAE应用程序，这些应用程序使用数据存储没有ide。。只有当您需要与您描述的某种担保进行交易时才是重要的。理解你什么时候需要它们是非常重要的，但你经常不需要。
数据存储在megastore之上实现，这在深度在本文中。简而言之，它在每个实体组中使用多版本并发控制，并且 Paxos ，以便跨数据中心进行复制，这两项功能都有助于潜艇写入。我不知道数据存储区中是否有潜艇写入频率的公开号码，但如果存在，则使用这些条款和数据存储邮件列表的搜索应找到它们。
<亚马逊的S3并不是一个真正的可比系统;它比分布式数据库更像CDN。亚马逊的SimpleDB具有可比性。它最初只提供了最终一致性，并最终添加了他们称之为非常有限的一种交易有条件写入，但它没有真正的事务。其他NoSQL数据库（redis，mongo，couchdb等）在交易和一致性方面有着不同的变化。基本上，分布式数据库在规模，事务广度以及一致性保证的强度。这一点最为人所熟知的是埃里克啤酒厂的 CAP定理，其中说，折中的三个轴是一致的，可用性和分区容限。
The Google App Engine documentation contains this paragraph:
Note: If your application receives an exception when committing a transaction, it does not always mean that the transaction failed. You can receive DatastoreTimeoutException, ConcurrentModificationException, or DatastoreFailureException exceptions in cases where transactions have been committed and eventually will be applied successfully. Whenever possible, make your Datastore transactions idempotent so that if you repeat a transaction, the end result will be the same.
Wait, what? It seems like there's a very important class of transactions that just simply cannot be made idempotent because they depend on current datastore state. For example, a simple counter, as in a like button. The transaction needs to read the current count, increment it, and write out the count again. If the transaction appears to "fail" but doesn't REALLY fail, and there's no way for me to tell that on the client side, then I need to try again, which will result in one click generating two "likes." Surely there is some way to prevent this with GAE?
Edit:
it seems that this is problem inherent in distributed systems, as per non other than Guido van Rossum -- see this link:
app engine datastore transaction exception
So it looks like designing idempotent transactions is pretty much a must if you want a high degree of reliability.
I was wondering if it was possible to implement a global system across a whole app for ensuring idempotency. The key would be to maintain a transaction log in the datastore. The client would generated a GUID, and then include that GUID with the request (the same GUID would be re-sent on retries for the same request). On the server, at the start of each transaction, it would look in the datastore for a record in the Transactions entity group with that ID. If it found it, then this is a repeated transaction, so it would return without doing anything.
Of course this would require enabling cross-group transactions, or having a separate transaction log as a child of each entity group. Also there would be a performance hit if failed entity key lookups are slow, because almost every transaction would include a failed lookup, because most GUIDs would be new.
In terms of the additional $ cost in terms of additional datastore interactions, this would probably still be less than if I had to make every transaction idempotent, since that would require a lot of checking what's in the datastore in each level.
dan wilkerson, simon goldsmith, et al. designed a thorough global transaction system on top of app engine's local (per entity group) transactions. at a high level, it uses techniques similar to the GUID one you describe. dan dealt with "submarine writes," ie the transactions you describe that report failure but later surface as succeeded, as well as many other theoretical and practical details of the datastore. erick armbrust implemented dan's design in tapioca-orm.
i don't necessarily recommend that you implement his design or use tapioca-orm, but you'd definitely be interested in the research.
in response to your questions: plenty of people implement GAE apps that use the datastore without idempotency. it's only important when you need transactions with certain kinds of guarantees like the ones you describe. it's definitely important to understand when you do need them, but you often don't.
the datastore is implemented on top of megastore, which is described in depth in this paper. in short, it uses multi-version concurrency control within each entity group and Paxos for replication across datacenters, both of which can contribute to submarine writes. i don't know if there are public numbers on submarine write frequency in the datastore, but if there are, searches with these terms and on the datastore mailing lists should find them.
amazon's S3 isn't really a comparable system; it's more of a CDN than a distributed database. amazon's SimpleDB is comparable. it originally only provided eventual consistency, and eventually added a very limited kind of transactions they call conditional writes, but it doesn't have true transactions. other NoSQL databases (redis, mongo, couchdb, etc.) have different variations on transactions and consistency.
basically, there's always a tradeoff in distributed databases between scale, transaction breadth, and strength of consistency guarantees. this is best known by eric brewer's CAP theorem, which says the three axes of the tradeoff are consistency, availability, and partition tolerance.
这篇关于GAE交易失败和幂等性的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
