如何使用 Python 在 Firestore 中下载大型集合而不会出现 503 错误? [英] How do i download a large collection in Firestore with Python without getting at 503 error?

问题描述

尝试使用 python 计算 firestore 集合中的文档数量.当我使用 db.collection('xxxx").stream() 时，我收到以下错误:

Trying to count the number of docs in a firestore collection with python. When i use db.collection('xxxx").stream() i get the following error:

 503 The datastore operation timed out, or the data was temporarily unavailable.

大约进行了一半.它工作正常.代码如下:

about half way through. It was working fine. Here is the code:

    docs = db.collection(u'theDatabase').stream()
    count = 0
    for doc in docs:
        count += 1
    print (count)

每次我在大约 73,000 条记录中收到 503 错误时.有谁知道如何克服 20 秒超时?

Every time I get a 503 error at about 73,000 records. Does anyone know how to overcome the 20 second timeout?

推荐答案

尝试使用递归函数批量检索文档并使其保持在超时范围内.以下是基于 delete_collections 片段的示例:

Try using a recursive function to batch document retrievals and keep them under the timeout. Here's an example based on the delete_collections snippet:

from google.cloud import firestore

# Project ID is determined by the GCLOUD_PROJECT environment variable
db = firestore.Client()


def count_collection(coll_ref, count, cursor=None):

    if cursor is not None:
        docs = [snapshot.reference for snapshot
                in coll_ref.limit(1000).order_by("__name__").start_after(cursor).stream()]
    else:
        docs = [snapshot.reference for snapshot
                in coll_ref.limit(1000).order_by("__name__").stream()]

    count = count + len(docs)

    if len(docs) == 1000:
        return count_collection(coll_ref, count, docs[999].get())
    else:
        print(count)


count_collection(db.collection('users'), 0)

这篇关于如何使用 Python 在 Firestore 中下载大型集合而不会出现 503 错误?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！