Firestore复合索引置换爆炸? [英] Firestore composite index permutation explosion?

问题描述

我正在考虑将30个约75万个文档集从mongo迁移到Google Firestore；但我遇到了复合索引问题.

I'm considering migrating a 30 collections, ~750k documents database from mongo to Google Firestore; but I'm running into problems with composite indexes.

我需要能够查询12个字段(即Field1 = A和Field7 = B和Field9 = C)-据我所知，每个组合都需要一个复合索引(?)

There's 12 fields I need to be able to query ad-hoc (ie. Field1=A and Field7=B and Field9=C) - as far as I can see I need a composite index for each combination(?)

这似乎不合比例；不仅不可能以编程方式创建索引；但是排列爆炸是真实的.每个文档都很大.下载大的子集并手动过滤不是可行的解决方案.

That does not seem to scale; not only is it impossible to programatically create indices; but the permutation explosion is real. Each document is fairly large; downloading a large subset and manually filter is not a viable solution.

我误解了索引还是在Firestore中根本不可能进行这些查询?

Am I misunderstanding indexes or are these queries simply not possible in Firestore?

推荐答案

firebase 如果您尝试执行以下任何查询，则几乎可以肯定会遇到问题.[更新]请查看本文的结尾，了解从2019年11月起可用的一些新选项.

firestore firebase If you are trying to do any of the following queries you will almost certainly run in to problems. [Update] Look at the end of this for some new options available from November 2019 onwards.

Field1 = A和Field2 = A

Field1=A and Field2=A

或

Field1 = A and Field2 = B and Field7 = D

Field1=A and Field2=B and Field7=D

如果要创建包含每个组合的复合索引，则复合索引的总数为200.

as the total number of composite indexes is 200, if you are trying to create composite indexes with each combination in.

但是，我的初步测试表明您可以执行以下操作.

However, my initial tests show you can do the following.

使用Field1到Field9来创建复合索引.

Create a composite index with Field1 to Field9 in.

这将用于满足包含这些字段的所有搜索(只要您在.where子句列表中没有其他字段).

This will be used to satisfy all the searches which contain these fields (as long as you do not have any other fields in the .where clause list).

如果同时使用order_by，则还必须符合DESCENDING或ASCENDING标准，但是由于您正在寻找完全匹配，因此这无关紧要.

You must also match the DESCENDING OR ASCENDING criteria if you are using order_by as well, but since you are looking at exact match this should not matter.

之所以可行，是因为firestore会智能地使用复合索引，并寻找一个包含所有字段的索引，而实际上包含更多字段的索引并不重要，因为.where子句没有为这些字段指定任何特定顺序，因此可以使用现有索引.

This works because firestore uses the composite indexes intelligently and looks for an index which has all the fields in and the fact it has more in does not matter as the .where clause does not specify any particular order for these fields and therefore the existing index can be used.

如果您要查找索引合并(靠近底部)，则会对此进行说明.

If you look for Index merging (close to the bottom) this is explained.

https://firebase.google.com/docs/firestore/query-data/index-overview

关于以编程方式创建和删除索引，现在比去年年底更有可能.

As for programatically creating and deleting indexes, it is now more possible than at the end of last year.

可以使用firebase工具 https://将现有的复合索引转储到文件中.firebase.google.com/docs/cli/

It is possible to dump the existing composite indexes into a file using the firebase tool https://firebase.google.com/docs/cli/

如果执行firebase初始化并选择firestore索引和规则，则将在当前文件夹/目录中创建一个名为firestore.indexes.json的文件，其中包括所有firestore复合索引以及所有排除项.我建议备份此文件，因为如果您弄乱了任何内容，它可以用来重新创建索引.

If you do a firebase init and select firestore indexes and rules, then this will result in a file called firestore.indexes.json being created in the current folder / directory which includes all the firestore composite indexes and all exclusions as well. I would recommend backing up this file as it can be used to recreate the indexes if you mess anything up.

可以将该文件添加到该命令，然后添加命令

This file can be added to and then the command

firebase部署-仅firestore:indexes

firebase deploy --only firestore:indexes

将在文件中添加所有组合索引(将创建新的组合索引，并且可以在firebase控制台的索引"选项卡中看到它们).现有的将保持不变.

will add all composite indexes in the file (new ones will be built and can be seen building in the firebase console in the indexes tab). Existing ones will be untouched.

这不会更改200个复合索引的限制..where子句列表(包括order_by字段)中也限制为100个字段.

This does not change the limit of 200 composite indexes. There is also a limit of 100 fields in a list of .where clauses including order_by fields.

您还可以使用gcloud SDK删除索引.请查看以下页面以获得说明.

You can also delete indexes using the gcloud SDK. Look at the following page for instructions.

https://cloud.google.com/sdk/install

确保在选项的安装列表中选择 beta .

Make sure that you select beta in the install list of options.

这是学习gcloud选项的良好入门

This is a good starter for learning about gcloud options

https://cloud.google.com/sdk/gcloud/reference/

以下页面描述了如何创建索引.

The following page describes how to create an index.

https://cloud.google.com/sdk/gcloud/reference/beta/firestore/indexes/composite/create

例如，

 gcloud beta firestore indexes composite create \
      --collection-group=Events \
      --field-config field-path=tags,array-config=contains \
      --field-config field-path=user_id,order=descending \
      --field-config field-path=timestamp,order=descending

有关更多可用标志的信息，请参见参考.

See the reference for more flags that can be used.

请注意，运行此命令将锁定命令行，直到索引创建完成为止，这可能需要5或10分钟，甚至更长的时间.如果您需要添加很多索引，则最好将其作为每个命令的后台命令运行，这样您就不会通过100个索引创建命令来进行单线程处理.

Just be aware that running this command will lock the command line until the index creation has complete and this could take 5 or 10 minutes or even longer. If you need to add a lot of indexes, it would be a good idea to run this as a background command for each one, so that you are not single threading through 100 index creation commands.

尽管可以仅通过ID删除复合索引，但是gcloud可以用于创建复合索引，也可以用于删除复合索引.

gcloud can then be used to create composite indexes and also used to delete composite indexes, although they can only be deleted by ID.

这可以通过使用命令找到

This can be found by using the command

gcloud beta firestore索引组合列表

gcloud beta firestore indexes composite list

+--------------+---------------------+-------------+-------+------------------------------+------------+--------------+
|     NAME     |   COLLECTION_GROUP  | QUERY_SCOPE | STATE |         FIELD_PATHS          |   ORDER    | ARRAY_CONFIG |
+--------------+---------------------+-------------+-------+------------------------------+------------+--------------+
| CICAgJjUt4gK | MyCollection        | COLLECTION  | READY | fieldStatus                  | ASCENDING  |              |
|              |                     |             |       | lastupdatedTimestamp         | DESCENDING |              |

要获取名称"值，请运行以下命令

To just get the Name value run the following command

gcloud beta firestore索引组合列表--format ="value(name)" --filter ="FIELD_PATHS:Field1"

gcloud beta firestore indexes composite list --format="value(name)" --filter="FIELD_PATHS:Field1"

这将提供复合索引名称的列表，然后可将其用于馈入删除命令，其中索引包括字段名称"Field1".

This will provide a list of composite Index names which can then be used to feed in to a delete command where the index includes the field name of "Field1".

可以使用以下删除命令，使用上面命令列出的名称(在此示例中为CICAgJjUt4gK)进行删除

Deletion can be done using the Name (CICAgJjUt4gK in this example) listed by the above command using the following deletion command

gcloud -q --account=[e-mail] --project = proj-a73464 beta Firestore索引复合删除CICAgJjUt4gK

gcloud -q --account=[e-mail] --project=proj-a73464 beta firestore indexes composite delete CICAgJjUt4gK

其中–account是针对Firebase项目注册的电子邮件地址，--project名称是您的项目名称，-q表示安静.这些选项必须位于命令的开头.

where --account is an email address registered against the firebase project and the --project name is your project name and the -q means quiet. These options MUST be at the beginning of the command.

索引删除的速度比创建索引的速度快得多，因此，如果您继续刷新Firestore控制台组合索引视图，您应该会看到计数下降.

The indexes delete much faster than they are created so if you keep refreshing the firestore console composite index view you should see the count going down.

要创建索引，请使用

gcloud beta firestore索引复合创建--collection-group = COLLECTION_GROUP --field-config = FIELD_CONFIG [--async] [GCLOUD_WIDE_FLAG…]

gcloud beta firestore indexes composite create --collection-group=COLLECTION_GROUP --field-config=FIELD_CONFIG [--async] [GCLOUD_WIDE_FLAG …]

用于创建.查找该命令的选项，但它们与delete命令类似.

for creation. Look up the options for this command but they will be similar to the delete command.

[2019年11月更新]

[Update November 2019]

Firestore现在允许使用"IN"和"array-contains-any".只能将10个值限制为10个值，因此不会满足OP所需的12个值，但是会满足很多要求，其中状态值可以是5个值之一，等等.

Firestore now allows "IN" and "array-contains-any" to be used. It is limited to 10 values for either of them so would not address the 12 needed by the OP, but would address a lot of requirements where status values can be one of 5 values etc.

此博客讨论了新功能.

https://firebase.googleblog.com/2019/11/cloud-firestore-now-supports-in-queries.html

这篇关于Firestore复合索引置换爆炸?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！