移动数据时数据流优于云功能的好处? [英] Benefits with Dataflow over cloud functions when moving data?

问题描述

我对 GCP 比较陌生，刚刚开始在 GCP 上设置/评估我的组织架构.

I'm relatively new to GCP and just starting to setup/evaluate my organizations architecture on GCP.

场景:
数据将流入发布/订阅主题(高频率、低数据量).目标是将这些数据移动到大表中.据我了解，您可以使用触发主题的云功能或使用 Dataflow 来做到这一点.

Scenario:
Data will flow into a pub/sub topic (high frequency, low amount of data). The goal is to move that data into Big Table. From my understanding you can do that either with a having a cloud function triggering on the topic or with Dataflow.

现在我对云功能有了满意的体验，所以这将是我的选择.

Now I have previous experience with cloud functions which I am satisfied with, so that would be my pick.

我没有看到选择其中一个的好处.所以我的问题是什么时候选择这些产品?

I fail to see the benefit of choosing one over the other. So my question is when to choose what of these products?

谢谢

推荐答案

两种解决方案都可以.如果您的 pub/sub 流量增长到大量数据，数据流将更好地扩展，但 Cloud Functions 应该适用于少量数据；我会查看此页面(尤其是速率限制部分)以确保您适合 Cloud Functions:https://cloud.google.com/functions/quotas

Both solutions could work. Dataflow will scale better if your pub/sub traffic grows to large amounts of data, but Cloud Functions should work fine for low amounts of data; I would look at this page (especially the rate-limit section) to ensure that you fit within Cloud Functions: https://cloud.google.com/functions/quotas

要考虑的另一件事是，Dataflow 可以保证对您的数据进行一次性处理，因此 BigTable 中不会出现重复数据.Cloud Functions 不会开箱即用地为您执行此操作.如果您使用函数方法，那么您将需要确保 Pub/Sub 消息一致地确定写入哪个 BigTable 单元格；这样，如果函数被重试多次，相同的数据将简单地覆盖相同的 BigTable 单元格.

Another thing to consider is that Dataflow can guarantee exactly-once processing of your data, so that no duplicates end up in BigTable. Cloud Functions will not do this for you out of the box. If you go with a functions approach, then you will want to make sure that the Pub/Sub message consistently determines which BigTable cell is written to; that way, if the function gets retried several times the same data will simply overwrite the same BigTable cell.

这篇关于移动数据时数据流优于云功能的好处?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！