Firestore截止日期大集合超出了例外 [英] Firestore DeadlineExceeded exception for big collections

查看：55
本文介绍了Firestore截止日期大集合超出了例外的处理方法，对大家解决问题具有一定的参考价值，需要的朋友们下面随着小编来一起学习吧！

问题描述

我正尝试从Google Firestore中读取更大的收藏集，以进行测试和存档.尝试从集合中包含超过6k个文档的所有文档中遇到一些有趣的错误.

I'm trying to read bigger collections from Google Firestore for testing and archiving purposes. I'm hitting some interesting errors when I try to get all documents from collections with more than 6k documents in them.

我的第一次尝试是使用Python google-cloud-firestore (版本0.30.0)库.

My first try was using the Python google-cloud-firestore (version 0.30.0) library.

source_client = firestore.Client()
source = source_client.collection(collection)
source_data = source.get()

counter = 0
for f in source_data:
    app.logger.info(f.id)
    counter += 1
    if counter % 100 == 0:
        app.logger.info('%s %d', datetime.now(), counter)

    app.logger.info('%s Finally read all %d documents', datetime.now(), counter)

哪个给出以下输出:

INFO:flask.app:2018-11-08 09:49:03.923795 6400  
INFO:flask.app:2018-11-08 09:49:04.115410 6500  
... 
INFO:flask.app:2018-11-08 09:49:03.923795 6400
INFO:flask.app:2018-11-08 09:49:04.115410 6500
WARNING:flask.app:2018-11-08 09:49:04.128478 copy brocken by exception
Traceback (most recent call last):
  File "/home/carsten/projects/transfertool/venv/lib/python3.6/site-packages/flask/app.py", line 2309, in __call__
    return self.wsgi_app(environ, start_response)
  File "/home/carsten/projects/transfertool/venv/lib/python3.6/site-packages/flask/app.py", line 2295, in wsgi_app
    response = self.handle_exception(e)
  File "/home/carsten/projects/transfertool/venv/lib/python3.6/site-packages/flask/app.py", line 1741, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/home/carsten/projects/transfertool/venv/lib/python3.6/site-packages/flask/_compat.py", line 35, in reraise
    raise value
  File "/home/carsten/projects/transfertool/venv/lib/python3.6/site-packages/flask/app.py", line 2292, in wsgi_app
    response = self.full_dispatch_request()
  File "/home/carsten/projects/transfertool/venv/lib/python3.6/site-packages/flask/app.py", line 1815, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/home/carsten/projects/transfertool/venv/lib/python3.6/site-packages/flask/app.py", line 1718, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/home/carsten/projects/transfertool/venv/lib/python3.6/site-packages/flask/_compat.py", line 35, in reraise
    raise value
  File "/home/carsten/projects/transfertool/venv/lib/python3.6/site-packages/flask/app.py", line 1813, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/carsten/projects/transfertool/venv/lib/python3.6/site-packages/flask/app.py", line 1799, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/home/carsten/projects/transfertool/firestore/transfertool/main.py", line 142, in transfer
    count_collection(source_collection)
  File "/home/carsten/projects/transfertool/firestore/transfertool/main.py", line 94, in count_collection
    for f in source_collection.offset(1000).get():
  File "/home/carsten/projects/transfertool/venv/lib/python3.6/site-packages/google/cloud/firestore_v1beta1/query.py", line 588, in get
    for index, response_pb in enumerate(response_iterator):
  File "/home/carsten/projects/transfertool/venv/lib/python3.6/site-packages/google/api_core/grpc_helpers.py", line 83, in next
    six.raise_from(exceptions.from_grpc_error(exc), exc)
  File "<string>", line 3, in raise_from
    # Permission is hereby granted, free of charge, to any person obtaining a copy
google.api_core.exceptions.DeadlineExceeded: 504 Deadline Exceeded

这似乎是由配额引起的.即使我不能在此处看到它.这似乎是基于时间的，因为当我在元素之间进行少量睡眠时，吞吐量会降低，并在约50秒后获得异常.

Which seems to be caused by a quota. Even if I cannot see it here. It seems to be time based because when I run with small sleeps in between elements I get less throughput and get the exception after ~50s.

对于这个问题，该库中有一个分页部分.由于我的应用程序不必关心我尝试传输的数据类型，因此无法使用 start_after 接口，但是仍然存在一个偏移接口，至少可以用它批量读取.

For exactly this problem there is a paging part in this library. As my application should not care about what kind of data I try to transfer I cannot use the start_after interface but there is still an offset interface with which I can at least read in batches.

for f in source_collection.offset(last_read_offset).get():

只要 last_read_offset 低于1001，就会给出正确的结果.如果我以1000的偏移量开始，直到得到 google.api_core.exceptions.DeadlineExceeded后，我才能得到结果.异常.但是当我从更大的东西开始时，我得到:

Which gives me correct results as long as the last_read_offset is below 1001. If I start with an offset of 1000 I can get results until I get the google.api_core.exceptions.DeadlineExceeded exception from above. But when I start with something bigger I get:

Traceback (most recent call last):
  File "/home/carsten/projects/transfertool/venv/lib/python3.6/site-packages/flask/app.py", line 2309, in __call__
    return self.wsgi_app(environ, start_response)
  File "/home/carsten/projects/transfertool/venv/lib/python3.6/site-packages/flask/app.py", line 2295, in wsgi_app
    response = self.handle_exception(e)
  File "/home/carsten/projects/transfertool/venv/lib/python3.6/site-packages/flask/app.py", line 1741, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/home/carsten/projects/transfertool/venv/lib/python3.6/site-packages/flask/_compat.py", line 35, in reraise
    raise value
  File "/home/carsten/projects/transfertool/venv/lib/python3.6/site-packages/flask/app.py", line 2292, in wsgi_app
    response = self.full_dispatch_request()
  File "/home/carsten/projects/transfertool/venv/lib/python3.6/site-packages/flask/app.py", line 1815, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/home/carsten/projects/transfertool/venv/lib/python3.6/site-packages/flask/app.py", line 1718, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/home/carsten/projects/transfertool/venv/lib/python3.6/site-packages/flask/_compat.py", line 35, in reraise
    raise value
  File "/home/carsten/projects/transfertool/venv/lib/python3.6/site-packages/flask/app.py", line 1813, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/carsten/projects/transfertool/venv/lib/python3.6/site-packages/flask/app.py", line 1799, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/home/carsten/projects/transfertool/firestore/transfertool/main.py", line 144, in transfer
    count_collection(source_collection)
  File "/home/carsten/projects/transfertool/firestore/transfertool/main.py", line 94, in count_collection
    for f in source_collection.offset(1001).get():
  File "/home/carsten/projects/transfertool/venv/lib/python3.6/site-packages/google/cloud/firestore_v1beta1/query.py", line 599, in get
    raise ValueError(msg)
ValueError: Unexpected server response. All responses other than the first must contain a document. The response at index 1 was
read_time {
  seconds: 1541668338
  nanos: 420813000
}
skipped_results: 1

查看库代码，似乎后端正在发送一条消息，该消息被解释为无效.

Looking at the library code it seems that the backend is sending a message which is interpreted as invalid.

好的，也许我的代码或Python客户端库已被窃听.让我们尝试一下节点.

Okay maybe my code or the Python client library is bugged. Let's try with node.

const admin = require('firebase-admin');
admin.initializeApp({
    credential: admin.credential.applicationDefault()
});

var db = admin.firestore();
admin.firestore().settings( { timestampsInSnapshots: true })
var counter = 0

console.log('Read collection')
db.collection(collection).get()
    .then(querySnapshot => {
        querySnapshot.forEach(documentSnapshot => {
            counter++;
        });
        console.log(counter)
    })
    .catch( error => {
        console.log(error)
});

哪怕超时明显超过60秒，它也与python库相同.

Which does the same as the python library even if the timeout is much more clearly 60s.

[2018-11-09T08:36:30.992Z] App listening on port 8080
[2018-11-09T08:36:30.993Z] Press Ctrl+C to quit.
[2018-11-09T08:36:37.390Z] Read collection
[2018-11-09T08:37:37.406Z] { Error: 4 DEADLINE_EXCEEDED: Deadline Exceeded
    at Object.exports.createStatusError (/home/carsten/projects/node_modules/grpc/src/common.js:87:15)
    at ClientReadableStream._emitStatusIfDone (/home/carsten/projects/node_modules/grpc/src/client.js:235:26)
    at ClientReadableStream._readsDone (/home/carsten/projects/node_modules/grpc/src/client.js:201:8)
    at /home/carsten/projects/node_modules/grpc/src/client_interceptors.js:679:15
  code: 4,
  metadata: Metadata { _internal_repr: {} },
  details: 'Deadline Exceeded' }

有人有类似的经历或一个很好的提示如何继续吗?

Has anyone similar experiences or a good hint how to continue?

PS: exportDocument / importDocument 接口不够用，因为有时我们在读取后必须调整数据.而且我不知道Firestore会将哪种格式导出到Google Cloud Storage或如何进行转换.

PS: The exportDocument / importDocument interface is not enough as we sometimes have to adjust the data after reading. And I have no clue what kind of format Firestore is exporting to Google Cloud Storage or how to transform it.

为此，我尝试了golang api.

And for the heck of it i tried the golang api.

log.Println("Collecting data")
snapshotIter := client.Collection(collection.(string)).Documents(ctx)
defer snapshotIter.Stop()

if err != nil {
    log.Fatalln(err)
}

i := 0

for {
    _, err := snapshotIter.Next()

    if err == iterator.Done {
        break
    }
    if err != nil {
        log.Fatalln(err)
    }

    if i % 100 == 0{
        log.Println(i)
    }
    i++
}

log.Println("Done")

哪个会遇到与预期相同的超时.

Which runs into the same timeout as expected.

2018/11/12 15:01:20 Collecting data
2018/11/12 15:01:21 0
2018/11/12 15:01:21 100
2018/11/12 15:01:21 200
2018/11/12 15:01:21 300
2018/11/12 15:01:21 400
2018/11/12 15:01:22 500
2018/11/12 15:01:22 600
2018/11/12 15:01:22 700
....
2018/11/12 15:02:22 29800
2018/11/12 15:02:23 29900
2018/11/12 15:02:23 rpc error: code = DeadlineExceeded desc = The datastore operation timed out, or the data was temporarily unavailable.

但是除此之外，偏移量也可以正常工作:

But in addition the offset is working fine:

snapshotIter := client.Collection(collection.(string)).Offset(30000).Documents(ctx)

推荐答案

在Firebase支持团队的帮助下，我们能够确定python客户端api确实存在错误.下一个版本中有一个错误修复程序.很有可能它将使python库能够按documentid排序，因此使用 start_after().

After some help from the firebase support team we were able to figure out that there is indeed a bug with the python client api. There is a bugfix coming in one of the next releases. Most likely it will enable the python library to sort by documentid and therefore use start_after().

到目前为止，您有两种可能的解决方案:

Up until then you have two possible solutions:

  1. 使用其他字段进行排序，并使用 start_after()

使用node.js库进行分页，例如:

use the node.js library with paging like:

var db = admin.firestore();
admin.firestore().settings({ timestampsInSnapshots: true });
function readNextPage(lastReadDoc) {
  let query = db
    .collection(collection)
    .orderBy(admin.firestore.FieldPath.documentId())
    .limit(100);
}

这篇关于Firestore截止日期大集合超出了例外的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！

查看全文
相关文章
Python最新文章
热门教程
热门工具
登录 关闭
扫码关注1秒登录
发送“验证码”获取 | 15天全站免登陆