Python Google Cloud Firestore错误504截止日期已超过 [英] Python Google cloud firestore error 504 Deadline Exceeded

问题描述

我在python中有一个fore firestore函数，其中我为一个集合的所有用户执行了一个for循环，然后进入另一个集合以获取一些指标，然后在第一个集合中更新此指标.

我运行了该函数，但是在执行过程中的某些时候，该函数中断了，给我这个错误:

  _Rendezvous Traceback(最近一次通话最后一次)〜\ Anaconda3 \ envs \ work \ lib \ site-packages \ google \ api_core \ grpc_helpers.py in next(self)78试试:--->79返回six.next(self._wrapped)80除了grpc.RpcError作为EXC:〜\ Anaconda3 \ envs \ work \ lib \ site-packages \ grpc \ _channel.py in __next __(self)363 def __next __(self):->364返回self._next()365_next(self)中的〜\ Anaconda3 \ envs \ work \ lib \ site-packages \ grpc \ _channel.py346其他:->347提高自我348而True:_Rendezvous:< _RPC的_Rendezvous终止于:状态= StatusCode.DEADLINE_EXCEEDED详细信息=超过最后期限"debug_error_string ="{" created:" @ 1570660422.708000000，" description:"收到来自对等ipv4的错误:216.58.202.234:443，" file:" src/core/lib/surface/call.cc，"file_line:1052，" grpc_message:"超过期限，" grpc_status:4}">上面的异常是以下异常的直接原因:DeadlineExceeded Traceback(最近一次通话结束)< ipython-input-20-05c9cefdafb4>在< module>中---->1个update_collection__persons()< ipython-input-19-6e2bdd597a6e>在update_collection__persons()中10 counter_secs = 011--->person_docs中的person_doc为12:13人_dict = person_doc.to_dict()14 last_updated = person_dict ['last_updated']流中的〜\ Anaconda3 \ envs \ work \ lib \ site-packages \ google \ cloud \ firestore_v1 \ query.py(自我，交易)766)767->768用于response_iterator中的响应:第769章没钱了第770章〜\ Anaconda3 \ envs \ work \ lib \ site-packages \ google \ api_core \ grpc_helpers.py in next(self)79返回six.next(self._wrapped)80除了grpc.RpcError作为EXC:--->81 six.raise_from(except.from_grpc_error(exc)，排除)8283#支持Python 2/3所需的别名.〜\ Anaconda3 \ envs \ work \ lib \ site-packages \ six.py在raise_from(value，from_value)中超过截止日期:超过504截止日期 

我一直在寻找解决方案，没有太多信息，在这里我发现了一个类似的问题:> https://github.com/googleapis/google-cloud-python/issues/8933

因此，我尝试使用此代码，但无法正常工作.这是我的功能:

  def update_collection__persons():人员= db.collection(u'collections__persons')person_docs = person.stream()counter_secs = 0对于person_docs中的person_doc:person_dict = person_doc.to_dict()last_updated = person_dict ['last_updated']last_processed = person_dict ['last_processed']dt_last_updated = datetime(1、1、1)+ timedelta(微秒= last_updated/10)dt_last_processed = datetime(1、1、1)+ timedelta(微秒= last_processed/10)如果dt_last_processed<dt_last_updated:order = db.collection(u'collection__orders').where(u'email'，u'=='，person_dict ['email'])orders_docs = orders.stream()sum_price = 0计数= 0date_add_list = []对于orders_docs中的order_doc:order_dict = order_doc.to_dict()sum_price + = order_dict ['total_price']数+ = 1date_add_list.append(order_dict ['dateAdded'])如果计数>0:data = {'metrics':{'LTV':sum_price，'AOV':sum_price/count，'Quantity_orders':计数，"first_order_date":分钟(date_add_list)，'last_order_date':max(date_add_list)}，'last_processed':int((datetime.utcnow()-datetime(1，1，1)).total_seconds()* 10000000)}db.collection(u'collection__persons').document(person_dict ['email']).set(data，merge = True) 

我创建了一个counter_secs只是为了查看该函数是否总是在同一查询中中断，而事实并非如此.

运行该功能后，如果我看到其中一些用户是随机用户，我也已经更新了他们的数据，因此它可以正常工作，但是在某些时候中断了

解决方案

persons.stream()的超时时间为60秒.与其在流式传输时处理每个文档，不如尝试先获取所有文档:

  person_docs = [persons.stream()中快照的快照] 

如果文档数量超过60秒内无法获取的大小，请尝试使用递归函数

I have been looking up for a solution, there is not much information, here I found a similar problem: https://github.com/googleapis/google-cloud-python/issues/8933

So I tried to use this code but is not working. This is my function:

def update_collection__persons():   
    persons = db.collection(u'collections__persons')
    person_docs = persons.stream()


    counter_secs = 0

    for person_doc in person_docs:
        person_dict = person_doc.to_dict()
        last_updated = person_dict['last_updated']
        last_processed = person_dict['last_processed']
        dt_last_updated = datetime(1, 1, 1) + timedelta(microseconds=last_updated/10)
        dt_last_processed = datetime(1, 1, 1) + timedelta(microseconds=last_processed/10)

        if dt_last_processed < dt_last_updated:
            orders = db.collection(u'collection__orders').where(u'email', u'==', person_dict['email'])
            orders_docs = orders.stream()

            sum_price = 0
            count = 0
            date_add_list = []

            for order_doc in orders_docs:
                order_dict = order_doc.to_dict() 
                sum_price += order_dict['total_price']
                count +=1
                date_add_list.append(order_dict['dateAdded'])
            if count > 0:
                data = {'metrics': {'LTV': sum_price,
                                    'AOV': sum_price/count,
                                    'Quantity_orders': count,
                                    'first_order_date': min(date_add_list),
                                    'last_order_date': max(date_add_list)},
                         'last_processed': int((datetime.utcnow() - datetime(1, 1, 1)).total_seconds() * 10000000)}

                 db.collection(u'collection__persons').document(person_dict['email']).set(data, merge = True)

I have created a counter_secs just to see if the function is breaking always in the same query but it is not.

Also after run the function if I see random users for some of them I have update their data, so it is working but breaking in some point

解决方案

There's a 60 second timeout for persons.stream(). Instead of processing each document as you stream, try fetching all the documents upfront:

person_docs = [snapshot for snapshot in persons.stream()]

If you have more documents than you can fetch in 60 seconds, try a recursive function like in this answer.

Same for the orders:

orders_docs = [snapshot for snapshot in orders.stream()]

这篇关于Python Google Cloud Firestore错误504截止日期已超过的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！