使用命令将数据库(具有嵌套文档)从Cloud Firestore导出到Bigquery [英] Export database(having nested documents) from cloud firestore to Bigquery using command

问题描述

我正在尝试将云Firestore数据导出到bigquery中以执行sql操作.

I am trying to export cloud firestore data into bigquery to do sql operations.

1. [使用]将云Firestore导出到云存储( https://cloud.google.com/firestore/docs/manage-data/export-import ) gcloud beta firestore导出gs://htna-3695c-storage --collection-ids ='users'，'feeds'
2. 按照 https://cloud.google.com/bigquery/docs/loading-data-cloud-firestoreto 从bigquery导入.

1. Exported cloud firestore to cloud storage [using] (https://cloud.google.com/firestore/docs/manage-data/export-import) gcloud beta firestore export gs://htna-3695c-storage --collection-ids='users','feeds'
2. Followed https://cloud.google.com/bigquery/docs/loading-data-cloud-firestoreto import from bigquery.

我们有2个收藏集:Users&馈入云Firestore.我已经成功导出了提要集合，但是无法导出用户集合.从存储导入数据到bigquery时出现错误

We have 2 collections: Users & Feeds in the cloud firestore. I have successfully exported feeds collection but am not able to export users collection. I am getting an error while importing data from storage to bigquery

错误:意外的属性名称联系人".我们在集合用户中有联系人"字段.此联系人字段的类型为地图".

Error: unexpected property name 'Contacts'. we have contacts field in the collection users. this contacts field is of type 'Map'.

我也尝试了命令行.下面是导出bigquery的命令.

I also tried the command line. Below is the command to export bigquery.

** bq --location = US load --source_format = DATASTORE_BACKUP myproject_Dataset.users gs://myproject-storage/2019-04-19T13:29:28_75338/all_namespaces/kind_users/all_namespaces_kind_users.export_metadata **

在这里我也遇到同样的错误:

here also I got the same error:

意外的属性名称"联系人".

'unexpected property name 'Contacts'.

我想添加投影字段以仅导出指定的字段，如下所示 ** bq --location = US load --projection_fields = [Coins，Referral，Profile] --source_format = DATASTORE_BACKUP myproject_Dataset.users gs://myproject-storage/2019-04-19T13:29:28_75338/all_namespaces/kind_users/all_namespaces_kind_users.export_metadata **

I thought to add projection fields to export only specified fields some thing like below **bq --location=US load --projection_fields=[Coins,Referral,Profile] --source_format=DATASTORE_BACKUP myproject_Dataset.users gs://myproject-storage/2019-04-19T13:29:28_75338/all_namespaces/kind_users/all_namespaces_kind_users.export_metadata**

在这里我也得到了错误:

here also I got the error:

正在等待> bqjob_r73b7ddbc9398b737_0000016a4909dd27_1 ...(0s)当前状态:完成
加载操作中的BigQuery错误:处理作业'myproject:bqjob_r73b7ddbc9398b737_0000016a4909dd27_1'时出错:内部错误发生，请求无法完成.错误:4557051

Waiting on >bqjob_r73b7ddbc9398b737_0000016a4909dd27_1 ... (0s) Current status: DONE
BigQuery error in load operation: Error processing job 'myproject:bqjob_r73b7ddbc9398b737_0000016a4909dd27_1': An internal error occurred and the request could not be completed. Error: 4550751

任何人都可以让我知道如何解决这些问题吗?

Can anyone please let me know how to fix these issues?

提前谢谢您. Firestore Db的图片

推荐答案

您的某些文档似乎有问题.根据Firestore的BigQuery导入的限制数据，则必须具有少于10,000个唯一字段名称的架构.在您的联系人架构中，您使用联系人的姓名作为键，这种设计可能会产生大量的字段名.您需要检查其他文档是否正在发生这种情况.

It seems to be an issue with some of your documents. According to the limitations of the BigQuery import of Firestore data, you must have a schema with less than 10,000 unique field names. In your Contacts schema, you are using the contacts' names as keys, that design is likely to produce a big amount of field names. You would need to check if this is happening to your other documents.

作为解决方法，您可以从以下位置更改设计(在加载过程的某个阶段):

As a workaround, you could change the design (at some stage of the loading process) from:

"Contacts" : {
    "contact1" : "XXXXX",
    "contact2" : "YYYYY",
}

收件人:

"Contacts" : [
    {
        "name" : "contact1",
        "number" : "XXXXX"
     },
     {
        "name" : "contact2",
        "number" : "YYYYY"
     },
]

此架构将大大减少field_name的数量，这将使其更易于从BigQuery中进行操作.

This schema would reduce drastically the number of field_names which will make it easier to manipulate from BigQuery.

这篇关于使用命令将数据库(具有嵌套文档)从Cloud Firestore导出到Bigquery的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！