管理数据存储区查看器中的App Engine查询返回的结果与编程式查询不同 [英] App Engine query in admin datastore viewer returning different results than programmatic query

问题描述

我很失望。

今天我注意到我的一些数据我认为应该出现在我的产品appengine应用程序中并没有显示出来。我通过远程控制台连接到应用程序并手动运行查询。果然，看起来我只有我想要看到的101行中有15行。

然后我在appengine.google.com上访问了我的管理控制台，然后启动使用以下查询的数据存储区查看器：

  SELECT * FROM Assignment where game = KEY（'Game'，'201212-foo' ）和player = KEY（'Player'，'player-mb'）
  

我看到的结果是20个结果的第一页。我翻阅了这些结果，并能够看到所有101个实体。万岁！我的数据仍然存在。但为什么我不能通过db api访问它？ （注意：我已经尝试通过memcache查看器清除memcache，即使这个查询不是手动memcached）

从远程控制台：

 > from google.appengine.ext.db import GqlQuery 
> GqlQuery（SELECT * FROM Assignment WHERE game = KEY（'Game'，'201212-foo'）and player = KEY（'Player'，'player-mb'））。count（）
 15 
  

远程控制台同意应用程序本身，它似乎只能看到预期的101行中的15行。

什么给予？

更新：

我怀疑这可能是一个索引问题。如果我为其中一个缺失的行发出get_by_key_name，它随后会显示在db api查询中。

 > GqlQuery（SELECT * FROM Assignment WHERE game = KEY（'Game'，'201212-foo'）and player = KEY（'Player'，'player-mb'））。count（）
 15 
> entities.Assignment.get_by_key_name（'201212-assignment-135.9'）
< entities.Assignment object at 0xa11eb6c> 
> count（）
 16 
  

那么我应该（或者可以）重建索引来解决这个问题吗？

UPDATE＃2 ：

我试图为这个查询建立一个完美的索引，即使查询使用刚构建的索引（通过query.index_list（）），结果仍然仅限于通过数据存储区查看器可用的一小部分项目。令人啼笑皆非的是，它实际上是一个不同于以前的索引（20项vs 15项）的子集。因此，现在添加额外的过滤条件会返回另外5行。如此愚蠢。

所有索引都声称是服务，所以不应该有任何理由指出这些索引太远了。

UPDATE＃3 ：

有时，使用我的新索引，我会得到正确的答案： p>

 > GqlQuery（SELECT * FROM Assignment WHERE game = KEY（'Game'，'201212-foo'）and player = KEY（'Player'，'player-mb'）and user ='zee'）。count（）
 101 
  

然而，如果我发出这个查询10次，它会返回'bad'结果约有一半时间：

 > GqlQuery（SELECT * FROM Assignment WHERE game = KEY（'Game'，'201212-foo'）and player = KEY（'Player'，'player-mb'）and user ='zee'）。count（）
 16 
  

所以，也许它是我碰到的一个糟糕/背后的bigtable副本问题一半的时间，或其他完全不透明的，我们不会得到答案（appengine 状态确实会列出今天的服务中断），但我有一种感觉，这将会自行修复。如果是，请再次更新。

最终更新： ，当我今天早上醒来时，我的应用程序（和手动查询）现在可以看到数据的一致，正确的视图。仍然会喜欢为什么发生这种情况的答案，但直到我得到这个结论时，我才会把它记录下来，直到内部的Google bigtable怪异。

我提交了这个问题反对appengine，看看我能否得到知情人士的回答。

对于HRD应用程序，这是按预期工作的。 App Engine高复制数据存储（HRD）将数据同步存储在多个数据中心中。但是，从提交写入到延迟到所有数据中心都可见为止，意味着跨多个实体组（非祖先查询）的查询只能保证最终一致结果。 [b]

在您的具体情况中，应用程序和管理控制台数据存储区查看器的结果之间的差异仅仅是因为它们很可能正在从不同的数据存储区服务器读取数据不同的一致性。

如果您需要一个一致的数据视图，我建议仔细阅读文章为强一致性构造数据

[1] https://developers.google.com/appengine/docs/java/datastore/structuring_for_strong_consistency

I'm flummoxed.

I noticed today that some data I thought should be present in my production appengine app wasn't showing up. I connected to the app via the remote console and ran the queries manually. Sure enough it looked like I only had 15 of the 101 rows I was expecting to see.

Then I went to my admin console at appengine.google.com and fired up the datastore viewer with the following query:

SELECT * FROM Assignment where game = KEY('Game', '201212-foo') and player = KEY('Player', 'player-mb')

The result I see is the first page of 20 results. I page through those results, and am able to see all 101 entities. HOORAY! My data is still there. BUT why then can't I access it via the db api? (NOTE: I've already tried clearing memcache via the memcache viewer, even though this particularly query isn't manually memcached)

From the remote console:

> from google.appengine.ext.db import GqlQuery
> GqlQuery("SELECT * FROM Assignment WHERE game = KEY('Game', '201212-foo') and player = KEY('Player', 'player-mb')").count()
15

The remote console agrees with the app itself, which only seems to be able to see 15 of the expected 101 rows.

What gives?

UPDATE:

I suspect this might be an indexing issue. If I issue get_by_key_name for one of the missing rows, it subsequently shows up in db api queries.

> GqlQuery("SELECT * FROM Assignment WHERE game = KEY('Game', '201212-foo') and player = KEY('Player', 'player-mb')").count()
15
> entities.Assignment.get_by_key_name('201212-assignment-135.9')
<entities.Assignment object at 0xa11eb6c>
> GqlQuery("SELECT * FROM Assignment WHERE game = KEY('Game', '201212-foo') and player = KEY('Player', 'player-mb')").count()
16

So should I (or can I) rebuild my indexes to remedy this problem?

UPDATE #2:

I attempted to build a perfect index for this query, and have just verified that even when the query does use the just-built index (via query.index_list()), the results are still only limited to a small subset of items available via the datastore viewer. Infuriatingly, it's actually a different subset than is available with the previous index (20 items vs 15 items). So now adding an additional filter term results in an additional 5 rows returned. So dumb.

All indexes claim to be "serving" so there shouldn't be any reason that the indexes are this far off.

UPDATE #3:

Sometimes, using my new index, I'll get the right answer:

> GqlQuery("SELECT * FROM Assignment WHERE game = KEY('Game', '201212-foo') and player = KEY('Player', 'player-mb') and user = 'zee'").count()
101

However if I issue this query 10 times, it comes back with the 'bad' results about half the time:

> GqlQuery("SELECT * FROM Assignment WHERE game = KEY('Game', '201212-foo') and player = KEY('Player', 'player-mb') and user = 'zee'").count()
16

So maybe its an issue of a bad/behind bigtable replica that I'm hitting half the time, or something else completely opaque that we won't get an answer to (appengine status does list a service disruption today), but I have a feeling that this will be fixed on its own. Will update again if it does.

FINAL UPDATE:

As I suspected, when I woke up this morning my app (and manual queries) now see a consistent, correct view of the data. Would still love an answer as to why this happened, but until I get that I'm going to chalk it up to internal Google bigtable weirdness.

I filed this issue against appengine to see if I can get an answer from someone in the know.

解决方案

For HRD applications, this is working as intended. App Engine High Replication Datastore (HRD) stores your data synchronously in multiple datacenters. However, the delay from the time a write is committed until it becomes visible in all datacenters means that queries across multiple entity groups (non-ancestor queries) can only guarantee eventually consistent results. [1]

In your specific case, the discrepancy between the results from your application and the Admin Console Datastore Viewer is just because they most likely are reading from different Datastore servers with different consistency.

If you require a consistent view of your data, I advise taking a closer look into the article "Structuring Data for Strong Consistency"

[1] https://developers.google.com/appengine/docs/java/datastore/structuring_for_strong_consistency

这篇关于管理数据存储区查看器中的App Engine查询返回的结果与编程式查询不同的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！