寻找提供与GAE数据库查询相匹配的项目的页面/项目计数/导航的想法/替代方案 [英] looking for ideas/alternatives to providing a page/item count/navigation of items matching a GAE datastore query

问题描述

我喜欢数据存储的简单性，可扩展性和易用性;并且新的 ndb 库中的增强功能非常棒。

据我了解数据存储最佳实践，当匹配查询的项目数很大时，不应编写代码来提供匹配查询结果的项目和/或页数;因为要做到这一点的唯一方法是检索所有的资源密集型结果。然而，在包括我们在内的许多应用程序中，常见的愿望是看到匹配项目的数量，并为用户提供导航到这些结果的特定页面的能力。数据存储分页问题由于需要解决 fetch（limit，offset = X）的限制而变得更加复杂，正如文章通过大数据集进行分页。为了支持推荐的方法，数据必须包含一个独特价值的列，可以按结果的显示方式进行排序。此列将为每个结果页面定义一个起始值;保存它，我们可以高效地获取相应的页面，允许根据请求导航到特定页面或下一页面。因此，如果您想要以多种方式显示结果，则可能需要维护多个此类列。

应该注意的是，从SDK v1开始。 3.1，推荐使用查询游标进行数据存储区分页。它们有一些限制，包括缺少对IN和！=过滤器运算符的支持。目前我们的一些重要查询使用 IN ，但我们会尝试使用或编写它们以用于查询游标。

按照建议的指导原则，可为用户提供（下一个）和（上一个）导航按钮，以及导航进行时的特定页面按钮。例如，如果用户按下（下一个） 3次，应用程序可以显示以下按钮，记住每个独特的起始记录或光标以保持导航高效：（Prev）（Page -1）（Page-2）（Page-3）（Page-4）（Next）。

有些人建议分开跟踪计数，但是当用户被允许在一组丰富的字段上查询时，这种方法是不实际的，这些字段会改变返回的结果。

我一般都在寻找关于这些问题的见解，以及以下问题：

1. 您在数据存储应用程序中提供了哪些查询结果的导航选项来解决这些限制？

如果为整个查询结果集中的用户提供高效的结果计数和页面导航
是一项优先事项，那么应该放弃使用数据存储区
来支持 GAE MySql解决方案现已提供。 在大表中是否有任何即将发生的变化体系结构还是
数据存储实现，它将为
计数查询结果提供额外的功能，以便高效地进行查询？

非常感谢您的帮助。

解决方案

<这一切都取决于你通常得到多少结果。例如。通过传递.count（）一个合适的限制，如果#items是例如< = 100，如果还有更多，则为很多。这听起来像你不能预先计算所有可能的计数，但至少你可以缓存它们，从而节省了很多数据存储操作。

使用NDB，最有效的方法可能是是使用fetch_page（）请求实体的第一页，然后使用结果光标作为count（）调用的起点;或者，您可能最好使用其异步工具同时运行第一页的fetch（）和count（）。如果您的查询不支持游标，第二个选项可能是您唯一的选择。大多数IN / OR查询当前不支持游标，但是如果您通过 __ key __ 命令来执行操作，则会执行此操作。

用户界面选项的条款，我认为这足以提供下一页和上一页的选项;可以提前跳过几页的GooooooogleUI很可爱，但我几乎从不使用它。 （要实现上一页，请颠倒查询的顺序，并使用与当前页面相同的游标，我很确定这是有效的。）

I like the datastore simplicity, scalability and ease of use; and the enhancements found in the new ndb library are fabulous.

As I understand datastore best practices, one should not write code to provide item and/or page counts of matching query results when the number of items that match a query is large; because the only way to do this is to retrieve all the results which is resource intensive.

However, in many applications, including ours, it is a common desire to see a count of matching items and provide the user with the ability to navigate to a specific page of those results. The datastore paging issue is further complicated by the requirement to work around limitations of fetch(limit, offset=X) as outlined in the article Paging Through Large Datasets. To support the recommended approach, the data must include a uniquely valued column that can be ordered in the way the results are to be displayed. This column will define a starting value for each page of results; saving it, we can fetch the corresponding page efficiently, allowing navigation to a specific or next page as requested. Therefore, if you want to show results ordered in multiple ways, several such columns may need to be maintained.

It should be noted that as of SDK v1.3.1, Query Cursors are the recommended way to do datastore paging. They have some limitations, including lack of support for IN and != filter operators. Currently some of our important queries use IN, but we'll try writing them using OR for use with query cursors.

Following the guidelines suggested, a user could be given a (Next) and (Prev) navigation buttons, as well as specific page buttons as navigation proceeded. For example if the user pressed (Next) 3 times, the app could show the following buttons, remembering the unique starting record or cursor for each to keep the navigation efficient: (Prev) (Page-1) (Page-2) (Page-3) (Page-4) (Next).

Some have suggested keeping track of counts separately, but this approach isn't practical when users will be allowed to query on a rich set of fields that will vary the results returned.

I'm looking for insights on these issues in general and the following questions specifically:

1. What navigational options of query results do you provide in your datastore apps to work around these limitations?

2. If providing users with efficient result counts and page navigation of the entire query result set is a priority, should use of the datastore be abandoned in favor of the GAE MySql solution now being offered.

3. Are there any upcoming changes in the big table architecture or datastore implementation that will provide additional capability for counting results of a query efficiently?

Many thanks in advance for your help.

解决方案

It all depends on how many results you typically get. E.g. by passing .count() a suitable limit you can provide an exact count if the #items is e.g. <= 100 and "many" if there are more. It sounds like you cannot pre-compute all possible counts, but at least you could cache them, thereby saving many datastore ops.

Using NDB, the most efficient approach may either be to request the first page of entities using fetch_page(), and then using the resulting cursor as a starting point for a count() call; or alternatively, you may be better off running the fetch() of the first page and the count() concurrently using its async facilities. The second option may be your only choice if your query does not support cursors. Most IN / OR queries don't currently support cursors, but they do if you order by __key__.

In terms of UI options, I think it's sufficient to offer next and previous page options; the "Gooooooogle" UI that affords skipping ahead several pages is cute but I almost never use it myself. (To implement "previous page", reverse the order of the query and use the same cursor you used for the current page. I'm pretty sure this is guaranteed to work.)

这篇关于寻找提供与GAE数据库查询相匹配的项目的页面/项目计数/导航的想法/替代方案的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！