GAE Appstats RPC时间轴图显示复杂NDB查询的长时间延迟 [英] GAE Appstats RPC Timeline graph shows long delays with complex NDB queries

问题描述

我从下面的我的制作应用搜索页面附加了Appstats。该页面需要约45秒才能通过AJAX加载结果。有大约100个实体。查询如下所示：

  qry_1 = X.query（ndb.AND（X.active_status ==active， X.property_3 == input_3，X.property_4 == input_4，X.property_5 == input_5，X.property_6.IN（input_6_list），X.property_20.IN（input_20_list）））
 record_list = qry_1.fetch（ ）
 
＃input_6_list包含〜5个字符串项目
＃input_20_list包含〜5个字符串项目
  

我无法弄清楚：

为什么RPC调用之间在下图中显示空白空间？它们表示什么？我怎样才能防止它们使我的网站无法使用

为什么查询需要很长时间才能完成

呼叫跟踪指向我的代码中的以下行：

  record_list = qry_1.fetch（）
  

Call Trace line显示它：

 <路径[0]> / m ain.py:6332 post（）
  

解决方案

自从没有真正改变你以前的问题。你的数据看起来非常相关。该模型是这样的，它强制查询执行大量的索引查找以满足AND和IN操作。简而言之，目前的模型永远不会缩放。它必须完全重新构造。

通过从查询开始创建新结构 - 以最快（换句话说，最简单）的方式输出所需数据查询参数和索引查找的最少数量？您可以连接所有旧的查询参数（属性），对结果进行散列，并最终以单个键在单个索引中进行查找。快速。然后从那里向后工作以存储您的记录，以便每个属性组合都散列到唯一键和相应的结果记录。您必须接受冗余（对不同的属性组合多次存储相同的结果），但这是NoSQL方式，而不是第三范式。

还有一些其他可能有用的建议用于在这些StackOverflow问题中重构数据：存储歌曲，艺术家和专辑数据和 How以存储文档结构。第二个答案：如果您的数据模型不灵活，请使用 Cloud SQL ，因为在小数据集上进行复杂查询会快得多。

I have attached the Appstats from my production app Search page below. The page takes ~45 seconds to load the results by AJAX. There are around 100 entities. The query is as shown below:

qry_1 = X.query(ndb.AND(X.active_status=="active", X.property_3==input_3, X.property_4==input_4, X.property_5==input_5, X.property_6.IN(input_6_list), X.property_20.IN(input_20_list))) 
record_list = qry_1.fetch() 

# input_6_list contains ~5 string items 
# input_20_list contains ~5 string items 

I am not able to figure out:
why are there empty spaces shown in the graph below between the RPC calls.. what do they signify.. how can I prevent them as they make my website unusable
why is the query taking so long to complete

The Call Trace points to the following line in my code:

record_list = qry_1.fetch() 

Call Trace line showing it:

  <path[0]>/main.py:6332 post() 

解决方案

The situation has not really changed since your previous question. Your data looks very relational. The model is such that it forces queries to perform large numbers of index lookups to satisfy AND and IN operations. In short, the current model will never scale. It must be completely restructured.

Create your new structure by starting at the query - what is the fastest (in other words simplest) way to just output the required data with the minimum number of query parameters and index lookups? You could concatenate all your old query parameters (properties), hash the result, and end up with a single key to look up in a single index. Fast. Then work backwards from there to storing your records such that every combination of properties hashes to a unique key and a corresponding result record. You must accept redundancy (storing the same result multiple times for different property combinations) but that is the NoSQL way, not Third Normal Form.

There are some other potentially useful suggestions for re-structuring your data in these StackOverflow questions: Storing song, artist and album data and How to store document structure.

Second answer: If you are inflexible on your data model, replace Datastore with Cloud SQL because that is much faster with complex queries on small data sets.

这篇关于GAE Appstats RPC时间轴图显示复杂NDB查询的长时间延迟的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！