使用 Avro/Parquet 的 Spark 作业中的最大行数 [英] Maximum size of rows in Spark jobs using Avro/Parquet
问题描述
我打算使用 Spark 来处理数据，其中 RDD 或 DataFrame 中的每个单独的元素/行有时可能会很大(高达几 GB).
I am planning to use Spark to process data where each individual element/row in the RDD or DataFrame may occasionally be large (up to several GB).
数据可能会存储在 HDFS 中的 Avro 文件中.
The data will probably be stored in Avro files in HDFS.
显然，每个执行程序都必须有足够的 RAM 来在内存中保存这些胖行"之一，还有一些要备用.
Obviously, each executor must have enough RAM to hold one of these "fat rows" in memory, and some to spare.
但是对于 Spark/HDFS 或常见序列化格式(Avro、Parquet、Sequence File...)的行大小是否还有其他限制?例如，这些格式的单个条目/行是否可以远大于 HDFS 块大小?
But are there other limitations on row size for Spark/HDFS or for the common serialisation formats (Avro, Parquet, Sequence File...)? For example, can individual entries/rows in these formats be much larger than the HDFS block size?
我知道已发布的 HBase 和 Cassandra 限制，但不知道 Spark...
I am aware of published limitations for HBase and Cassandra, but not Spark...
推荐答案
目前有一些与块大小相关的基本限制，对于正在使用的分区和 shuffle 块 - 两者都限制为 2GB，这是 ByteBuffer 的最大大小(因为它需要一个 int 索引，所以限制为 Integer.MAX_VALUE 字节).
There are currently some fundamental limitations related to block size, both for partitions in use and for shuffle blocks - both are limited to 2GB, which is the maximum size of a ByteBuffer (because it takes an int index, so is limited to Integer.MAX_VALUE bytes).
单个行的最大大小通常需要远小于最大块大小，因为每个分区通常会包含许多行，并且最大的行可能不会在分区之间均匀分布 - 如果一个分区偶然包含异常大量的大行，这可能会使其超过 2GB 限制，从而导致作业崩溃.
The maximum size of an individual row will normally need to be much smaller than the maximum block size, because each partition will normally contain many rows, and the largest rows might not be evenly distributed among partitions - if by chance a partition contains an unusually large number of big rows, this may push it over the 2GB limit, crashing the job.
见:
这些 Spark 问题的相关 Jira 票证:
Related Jira tickets for these Spark issues:
- https://issues.apache.org/jira/browse/SPARK-1476
- https://issues.apache.org/jira/browse/SPARK-5928
- https://issues.apache.org/jira/browse/SPARK-6235
这篇关于使用 Avro/Parquet 的 Spark 作业中的最大行数的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
