使用Apache Beam进行Dataflow批量加载时的性能问题 [英] Performance issues on Dataflow batch loads using Apache Beam

问题描述

我正在做一个数据流批量加载的性能基准测试，并发现与BigQuery命令行工具上的相同加载相比，加载速度太慢。

I was doing a performance benchmarking of dataflow batch loads and found that the loads were just too slow when compared against the same loads on Bigquery command line tool.

文件大小约为20 MB，拥有数百万条记录。我尝试了不同的机器类型，并在加载目标BQ表时大约加载时间为8分钟，在 n1-highmem-4 上获得了最佳加载性能。

The file size was around 20 MB with millions of records. I tried different machine types and got the best load performance on n1-highmem-4 with the approx load time of 8 minutes in loading the target BQ table.

当通过在命令行实用程序上运行BQ命令应用相同的表加载时，几乎不用花2分钟来处理和加载相同的数据量。
有关使用Dataflow作业的不良负载性能的任何见解？如何提高性能以使其与BQ命令行实用程序相媲美？

When the same table load was applied by running BQ command on the command-line utility, it hardly took 2 minutes to process and load the same volume of data. Any insights about this poor load performance using Dataflow jobs? How to improve the performance to make it comparable to BQ command line utility?

推荐答案

很可能，几分钟的时间都花在启动和关闭虚拟机上。如果你正在做一些可以直接使用BQ CLI完成的事情，那么为此目的使用Dataflow可能是过度的。但是，您可以更新您的问题并提供更多详细信息（例如您的代码和Dataflow作业ID） - 也许还有其他效率不高的问题。

Most likely, a few minutes are being spent on starting and shutting down VMs. If you're doing something that can directly be done using BQ CLI, then using Dataflow for that purpose is likely overkill. However, you can update your question with more details (e.g. your code and the Dataflow job id) - maybe there's something else inefficient going on.

这篇关于使用Apache Beam进行Dataflow批量加载时的性能问题的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！