Apache Beam-我应该了解编写高效数据处理管道的关键概念是什么? [英] Apache Beam - What are the key concepts for writing efficient data processing pipelines I should be aware of?
问题描述
我已经使用Beam一段时间了，我想知道编写有效和优化的Beam管道的关键概念是什么.
I've been using Beam for some time now and I'd like to know what are the key concepts for writing efficient and optimized Beam pipelines.
我有一点Spark背景，并且我知道我们可能更喜欢使用reduceByKey而不是groupByKey以避免改组并优化网络流量.
I have a little Spark background and I know that we may prefer to use a reduceByKey instead of a groupByKey to avoid shuffling and optimise network traffic.
Beam的情况也一样吗?
Is it the same case for Beam?
我会喜欢一些技巧或材料/最佳实践.
I'd appreciate some tips or materials/best pratices.
推荐答案
要考虑的一些项目:
-
文件管理器优先；尽可能在DAG中放置过滤器操作)
Filer first; place filter operations as high in the DAG as possible)
早点结合；如果可以选择何时合并，请尽早进行.
Combine early; If there is a choice as to when to combine, do this as early as possible.
如果可能的话，可以通过在大型滑动窗口之前使用较小的固定窗口来减小大型滑动窗口的影响. FixedWindow.of(1m)|结合| SlidingWindow.of(6小时)
If possible reduce the effect of large sliding windows by using smaller fixed windows before the large sliding window. FixedWindow.of(1m) | Combine | SlidingWindow.of( 6 Hour)
大多数跑步者将支持图融合，这在99％的时间内都是正确的选择.但是在大规模扇出变换的情况下，您应该中断融合.
Most runners will support graph fusion, which is the right thing 99% of the time. But in the case of a massive fanout transform you should break fusion.
- 选择提供良好性能的编码器(例如，在Java中)使用Proto或Avro编码器之类的东西，而不使用默认的Java序列化.
- 高级技巧:编码/解码是很大的开销来源.因此，如果您有一个较大的Blob，但只需要结构化的一部分，则可以选择性地仅解码该部分.
- 避免在每个元素级别使用Log.info，这很少有价值，并且是许多与性能相关的问题的根本原因.
- Understand the data set and the implications of Hot keys. Fields that are used as Keys which are Nullable are often culprits... Make use of parallelism hints if needed withFanOut
For keys in general
- 密钥太少:不好-难以分担工作负载，按密钥排序会影响性能
- 密钥太多:可能也很糟糕-开销开始增加.
高级按键提示:
- 有时您可以将键与元素{key，Window}的窗口结合使用，以帮助更多地分配工作
- 不是必需条件，但是如果您有能力并且想进入这种优化级别；瞄准〜O(10K)至O(100K)键.如果键空间大得多，则可以考虑使用散列在内部将键分开.如果键带有日期/时间信息，则此功能特别有用.在这种情况下，您可以免费重用过去不再使用的过去处理过的密钥.
-
使用选项标志可以轻松读取压缩文件，但是如果没有Offset TextIO，则无法分发此任务.如果您有非常大的文件要读取，请在启动管道之前读取解压缩文件，这样可以大大提高性能.还要看看使用压缩的Avro之类的格式.
Compressed files can be easily read with an option flag, however without an Offset TextIO can not distribute this task. If you have very large files to read uncompressing the file before starting the pipeline can provide a nice performance boost. Also look at using formats like compressed Avro.
-
BackPressure:光束流道设计为能够快速咀嚼平行工作.他们可以在许多计算机上增加许多线程来实现此目标.这很容易淹没外部系统，尤其是在进行每个元素的RPC调用时.如果无法扩展外部系统，请使用startBundle/finishBundle创建批处理，以帮助提高每秒的调用次数
BackPressure: Beam runners are designed to be able to rapidly chew through parallel work. They can spin up many threads across many machines to achieve this goal. This can easily swamp external systems, especially if you are making a per element RPC call. If the external system can not be made to scale, create batches using startBundle / finishBundle to help elevate the invocations per second
光速仍然是光速.:-)避免使用与工作人员相距较远的水槽和水源.
Speed of light, is well still the speed of light.. :-) Avoid using sinks and sources which are far apart from your workers.
- 利用Beam指标来度量您的管道.
这篇关于Apache Beam-我应该了解编写高效数据处理管道的关键概念是什么?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
