增加Google Cloud DataStore组合索引限制 [英] Increasing Google Cloud DataStore Composite index limit

问题描述

我使用谷歌应用程序引擎作为数据库的后端和数据存储区。链接 https://cloud.google.com/datastore/docs/concepts/limits 表示项目的最大组合索引数不能超过200.
我的项目中有大约130个组合索引，并且在未来某个时候会达到此限制。

200的限制对我来说似乎非常小。
假设我的项目中有5个模块，每个模块都有10种种类。在每种类型中，我有4个属性，我想索引（让我们称它们为prop1，prop2，prop3和prop4）。此外，每种种类都有一个名为creationTime的字段，该字段存储实体在数据存储中创建的时间。无论我应用0,1,2,3还是全部4个过滤器，我总是希望我的实体列表按照creationTime排序，最新的排在第一位。

这似乎对我来说是一个完全合理的场景。在这种情况下，对于每种种类，我将不得不定义下列组合索引：

pre $ < datastore-index kind = kind1ancestor =false>
< property name =prop1direction =asc/>
< property name =creationTimedirection =desc/>
< / datastore-index>
< datastore-index kind =kind1ancestor =false>
< property name =prop2direction =asc/>
< property name =creationTimedirection =desc/>
< / datastore-index>
< datastore-index kind =kind1ancestor =false>
< property name =prop3direction =asc/>
< property name =creationTimedirection =desc/>
< / datastore-index>
< datastore-index kind =kind1ancestor =false>
< property name =prop4direction =asc/>
< property name =creationTimedirection =desc/>
< / datastore-index>


因为有50种这样的类型，所以会有200个这样的索引。现在我知道如果我不按creationTime排序实体列表，我可以避免使用这些索引，但从用户的角度来看，这真的很糟糕。

有没有办法增加/克服这个限制？
我在这里错过了什么吗？
我是否需要限制我的查询？如果是，那么我如何才能实现相同的用户体验？
数据存储不适用于此类查询吗？我有什么选择吗？

解决方案

你不能增加限制，所以你应该看看你的数据模型。首先，让我们来澄清一下术语：你称之为实体的东西实际上叫做'种类'。实体是一种单独的记录。

检查你的种类，看看它们是否真的在语义上不同，或者它们实际上是非常相似（许多重叠的属性）。如果它们相似，则可以将它们全部放在同一类中，并添加一个属性以区分它们;让我们把它称为类型属性。

例如，不是为巨魔，僵尸和巫婆，你可以有一种叫做怪物。

现在，您的示例索引：

 < datastore-index kind =Entity1ancestor =false> 
< property name =prop1direction =asc/> 
< property name =creationTimedirection =desc/> 
< / datastore-index> 
  

会出现以下情况：

 < datastore-index kind =Masterancestor =false> 
< property name =typedirection =Entity1/> 
< property name =prop1direction =asc/> 
< property name =creationTimedirection =desc/> 
< / datastore-index> 
  

有什么好处，是过滤 prop1 并按> creationTime 进行排序只需要一个组合索引，无论类型的数量如何。因此，在您的50种示例中，而不是50种覆盖每种类型的复合索引，现在只有1种。

I am using google app engine for my backend and datastore as the DB. The link https://cloud.google.com/datastore/docs/concepts/limits indicates that the maximum number of composite indexes for a project cannot be more than 200. I have around 130 composite indexes in my project and would hit the limit sometime in the future.

Limit of 200 seems very less to me. Let's say I have 5 modules in my project and each module has 10 "kinds" each. In each of the kinds, I have 4 properties I want to index upon (let's call them prop1, prop2, prop3 and prop4). Also each of the "kinds" have a field called creationTime, which stores the time at which the entity was created in the datastore. Whether I apply 0, 1 ,2 ,3 or all 4 of the filters, I always want my list of entities to be sorted by creationTime with newest first.

This seems to me to be a perfectly reasonable scenario. In this case for each "kind", I would have to define the following composite indexes

<datastore-index kind="kind1" ancestor="false">
        <property name="prop1" direction="asc" />
        <property name="creationTime" direction="desc" />
</datastore-index>
<datastore-index kind="kind1" ancestor="false">
        <property name="prop2" direction="asc" />
        <property name="creationTime" direction="desc" />
</datastore-index>
<datastore-index kind="kind1" ancestor="false">
        <property name="prop3" direction="asc" />
        <property name="creationTime" direction="desc" />
</datastore-index>
<datastore-index kind="kind1" ancestor="false">
        <property name="prop4" direction="asc" />
        <property name="creationTime" direction="desc" />
</datastore-index>

Since there are 50 such kinds, there would be 200 such indexes. Now I know I can avoid these indexes if I don't sort the list of entities by creationTime, but I think that would be really bad from the user perspective.

So is there any way to increase / overcome the limit? Am I missing something here? Do I need to limit my queries ? If yes then how can I achieve the same user experience? Is datastore not meant for such queries? What options do I have here?

解决方案

You cannot increase the limit, so you should look at your data model instead.

First, let's clear up terminology: What you are calling 'entities' are really called 'kinds'. Entities are the individual records in a kind.

Review your kinds and see if they are really semantically different, or if they are actually very similar (many overlapping properties). If they are similar, you can have them all in the same kind and add a property to distinguish between them; let's call it the type property.

For example, rather than having separate kinds for trolls, zombies, and witches, you could have a single kind called monsters.

Now, your example index:

<datastore-index kind="Entity1" ancestor="false">
        <property name="prop1" direction="asc" />
        <property name="creationTime" direction="desc" />
</datastore-index>

Would be the following:

<datastore-index kind="Master" ancestor="false">
        <property name="type" direction="Entity1" />
        <property name="prop1" direction="asc" />
        <property name="creationTime" direction="desc" />
</datastore-index>

What is good about this, is filter prop1 and sort by creationTime only takes one composite index, regardless of the number of types. So in your example of 50 kinds, rather than 50 composite indexes to cover each kind, you now only have 1.

这篇关于增加Google Cloud DataStore组合索引限制的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！