从 Pub/Sub 订阅中提取的 Google Cloud Function 引发异常 - 已超过截止日期 [英] Google Cloud Function pulling from Pub/Sub subscription throws exception - Deadline Exceeded

问题描述

我在 Python 3.7 中有一个 Google Cloud Function，它以同步拉取模式从 Pub/Sub 订阅中读取数据.

I have a Google Cloud Function in Python 3.7 reading from a Pub/Sub subscription in synchronous pull mode.

在 1/hour 正常运行 24 小时后，它抛出了这个异常堆栈跟踪:

After running fine 1/hour for 24 hours, it threw this exception stack trace:

Traceback(最近一次调用最后一次):文件
"/env/local/lib/python3.7/site-packages/google/api_core/grpc_helpers.py",
第 57 行，在 error_remapped_callable
返回 callable_(*args, **kwargs) 文件/env/local/lib/python3.7/site-packages/grpc/_channel.py"，第 824 行，
在通话中
返回_end_unary_response_blocking(状态，调用，假，无)文件/env/local/lib/python3.7/site-packages/grpc/_channel.py"，行
726，在_end_unary_response_blocking
raise _InactiveRpcError(state) grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC 终止于:status =
StatusCode.DEADLINE_EXCEEDED details = "已超过截止日期"
debug_error_string =
"{"created":"@1580454091.145703535","description":"收到的错误来自
同行
ipv4:74.125.202.95:443","file":"src/core/lib/surface/call.cc","file_line":1056,"grpc_message":"截止日期
超过","grpc_status":4}"

Traceback (most recent call last): File
"/env/local/lib/python3.7/site-packages/google/api_core/grpc_helpers.py",
line 57, in error_remapped_callable
return callable_(*args, **kwargs) File "/env/local/lib/python3.7/site-packages/grpc/_channel.py", line 824,
in call
return _end_unary_response_blocking(state, call, False, None) File "/env/local/lib/python3.7/site-packages/grpc/_channel.py", line
726, in _end_unary_response_blocking
raise _InactiveRpcError(state) grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with: status =
StatusCode.DEADLINE_EXCEEDED details = "Deadline Exceeded"
debug_error_string =
"{"created":"@1580454091.145703535","description":"Error received from
peer
ipv4:74.125.202.95:443","file":"src/core/lib/surface/call.cc","file_line":1056,"grpc_message":"Deadline
Exceeded","grpc_status":4}"

上述异常是以下异常的直接原因:

The above exception was the direct cause of the following exception:

Traceback(最近一次调用最后一次):文件
"/env/local/lib/python3.7/site-packages/google/cloud/functions/worker.py",
第 346 行，在 run_http_function
结果 = _function_handler.invoke_user_function(flask.request) 文件
"/env/local/lib/python3.7/site-packages/google/cloud/functions/worker.py",
第 217 行，invoke_user_function
返回 call_user_function(request_or_event) 文件/env/local/lib/python3.7/site-packages/google/cloud/functions/worker.py"，
第 210 行，在 call_user_function
在迭代中返回 self._user_function(request_or_event) 文件/user_code/main.py"，第 39 行
response = sub.pull(sub_path, MAX_MESSAGES) 文件/env/local/lib/python3.7/site-packages/google/cloud/pubsub_v1/_gapic.py",
第 40 行，在
fx = lambda self, *a, **kw: Wrapped_fx(self.api, *a, **kw) # noqa File
"/env/local/lib/python3.7/site-packages/google/cloud/pubsub_v1/gapic/subscriber_client.py",
第 1005 行，拉中
请求，重试=重试，超时=超时，元数据=元数据文件/env/local/lib/python3.7/site-packages/google/api_core/gapic_v1/method.py"，
第 143 行，在 调用
返回 Wrapped_func(*args, **kwargs) 文件 "/env/local/lib/python3.7/site-packages/google/api_core/retry.py",
第 286 行，在 retry_wrapped_func
on_error=on_error，文件/env/local/lib/python3.7/site-packages/google/api_core/retry.py"，
第 184 行，在 retry_target
返回目标()文件/env/local/lib/python3.7/site-packages/google/api_core/timeout.py"，
第 214 行，在 func_with_timeout
返回 func(*args, **kwargs) 文件 "/env/local/lib/python3.7/site-packages/google/api_core/grpc_helpers.py",
第 59 行，在 error_remapped_callable
六.raise_from(exceptions.from_grpc_error(exc)，exc)文件"，第3行，在raise_from
google.api_core.exceptions.DeadlineExceeded:超过 504 截止日期

Traceback (most recent call last): File
"/env/local/lib/python3.7/site-packages/google/cloud/functions/worker.py",
line 346, in run_http_function
result = _function_handler.invoke_user_function(flask.request) File
"/env/local/lib/python3.7/site-packages/google/cloud/functions/worker.py",
line 217, in invoke_user_function
return call_user_function(request_or_event) File "/env/local/lib/python3.7/site-packages/google/cloud/functions/worker.py",
line 210, in call_user_function
return self._user_function(request_or_event) File "/user_code/main.py", line 39, in iteration
response = sub.pull(sub_path, MAX_MESSAGES) File "/env/local/lib/python3.7/site-packages/google/cloud/pubsub_v1/_gapic.py",
line 40, in
fx = lambda self, *a, **kw: wrapped_fx(self.api, *a, **kw) # noqa File
"/env/local/lib/python3.7/site-packages/google/cloud/pubsub_v1/gapic/subscriber_client.py",
line 1005, in pull
request, retry=retry, timeout=timeout, metadata=metadata File "/env/local/lib/python3.7/site-packages/google/api_core/gapic_v1/method.py",
line 143, in call
return wrapped_func(*args, **kwargs) File "/env/local/lib/python3.7/site-packages/google/api_core/retry.py",
line 286, in retry_wrapped_func
on_error=on_error, File "/env/local/lib/python3.7/site-packages/google/api_core/retry.py",
line 184, in retry_target
return target() File "/env/local/lib/python3.7/site-packages/google/api_core/timeout.py",
line 214, in func_with_timeout
return func(*args, **kwargs) File "/env/local/lib/python3.7/site-packages/google/api_core/grpc_helpers.py",
line 59, in error_remapped_callable
six.raise_from(exceptions.from_grpc_error(exc), exc) File "", line 3, in raise_from
google.api_core.exceptions.DeadlineExceeded: 504 Deadline Exceeded

这是怎么回事?这是意料之中的还是某些配置问题的结果?如果符合预期，应该如何处理?

What is this about? Is it to be expected or a result of some configuration problem? If to be expected, how should it be handled?

文档(查看源代码:https://googleapis.dev/python/pubsub/latest/subscriber/api/client.html )在 pull 上没有任何关于这是一个可能的例外.

The documentation ( view-source:https://googleapis.dev/python/pubsub/latest/subscriber/api/client.html ) on pull has nothing about this being a possible exception.

我在拉取完成后立即确认消息.我一次只允许执行一个函数.我有 600 秒的确认截止日期.一次提取的消息块似乎少于 100 条.如果这是关于未能确认消息，那么错误似乎可以做得更好.

I ack the messages immediately after the pull completes. I only permit one function execution at a time. I have a 600 second acknowledgement deadline. A block of messages pulled at one time seem to be less than 100 in number. If this is about failing to ack a message, it seems like the error could be done much better.

推荐答案

当订阅中没有要读取的消息时，客户端会引发此异常.这是来自最新 PubSub 库版本 >= 1.0.0 的 已知问题.如有必要，您可以降级到不存在此问题的版本 0.45.0.

This exception is raised by the client when there's no messages to read in the subscription. It is a known issue from the latest PubSub library versions >= 1.0.0. If necessary, you can downgrade to the version 0.45.0 where this issue was not present.

但是，作为一种解决方法，您可以捕获 DeadlineExceeded 异常并再次重试该操作.另外，根据禾芒的评论，这里有一个小可以添加到运行代码中的 monkeypatch，这可能有助于获得与版本 0.45.0 中相同的行为.

However, as a workaround you can catch the DeadlineExceeded exception and retry the operation again. Also, based on the comment of Hemang, here's a small monkeypatch that you can add to your running code, which might help to get the same behavior as in version 0.45.0.

from google.cloud.pubsub_v1.gapic import subscriber_client_config as sub_config
sub_config.config['interfaces']['google.pubsub.v1.Subscriber']['retry_params']['messaging']['initial_rpc_timeout_millis'] = 25000

最后，请记住，在使用同步拉取时，拥有许多未完成的拉取请求会有所帮助降低交付延迟，这反过来可能导致更高的延迟拉取请求(和 DeadlineExceeded 错误).虽然，如果延迟对应用程序至关重要，您可以考虑使用 StreamingPull

Finally, keep in mind that when using synchronous pull, having many outstanding pull requests helps lower the delivery latency, which in turn might result in higher latency pull requests (and DeadlineExceeded errors). Although, if latency is crucial for the application, you could consider using StreamingPull

这篇关于从 Pub/Sub 订阅中提取的 Google Cloud Function 引发异常 - 已超过截止日期的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！