我可以强制我的数据流管道中的一个步骤是单线程的(并且在一台机器上)吗? [英] Can I force a step in my dataflow pipeline to be single-threaded (and on a single machine)?

问题描述

我有一个管道，可以获取文件的 URL 并下载这些文件，为除标题之外的每一行生成 BigQuery 表行.

I have a pipeline that takes URLs for files and downloads these generating BigQuery table rows for each line apart from the header.

为了避免重复下载，我想根据以前下载的表检查 URL，如果 URL 不在此历史"表中，则仅继续存储该 URL.

To avoid duplicate downloads, I want to check URLs against a table of previously downloaded ones and only go ahead and store the URL if it is not already in this "history" table.

为此，我需要将历史记录存储在允许唯一值的数据库中，或者为此使用 BigQuery 可能更容易，但对表的访问必须严格串行.

For this to work I need to either store the history in a database allowing unique values or it might be easier to use BigQuery for this also, but then access to the table must be strictly serial.

我可以强制执行单线程执行(在一台机器上)来满足我的部分管道的要求吗?

Can I enforce single-thread execution (on a single machine) to satisfy this for part of my pipeline only?

(在此之后，我的 100 个 URL/文件中的每一个都适合在单独的线程上处理；每个单个文件产生 10000-10000000 行，因此在这一点上进行节流几乎肯定不会产生性能问题.)

(After this point, each of my 100s of URLs/files would be suitable for processed on a separate thread; each single file gives rise to 10000-10000000 rows, so throttling at that point will almost certainly not give performance issues.)

推荐答案

Beam 是为数据的并行处理而设计的，它试图明确阻止您进行同步或阻塞，除非使用一些内置原语，例如 组合.

Beam is designed for parallel processing of data and it tries to explicitly stop you from synchronizing or blocking except using a few built-in primitives, such as Combine.

听起来您想要的是一个过滤器，该过滤器仅在第一次看到时才发出元素(您的 URL).您可能可以使用内置的 不同的变换为此.此运算符使用每个键的组合按键(在本例中为您的 URL)对元素进行分组，然后仅在第一次看到时发出每个键.

It sounds like what you want is a filter that emits an element (your URL) only the first time it is seen. You can probably use the built-in Distinct transform for this. This operator uses a Combine per-key to group the elements by key (your URL in this case), then emits each key only the first time it is seen.

这篇关于我可以强制我的数据流管道中的一个步骤是单线程的(并且在一台机器上)吗?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！