在 Spark sc.newAPIHadoopRDD 中读取 2.7 GB 数据，有 5 个分区 [英] In Spark sc.newAPIHadoopRDD is reading 2.7 GB data the with 5 partitions

问题描述

我正在使用 spark 1.4，我正在尝试使用 sc.newAPIHadoopRDD 从 Hbase 读取数据以读取 2.7 GB 数据，但为此阶段创建了 5 个任务，处理它需要 2 t0 3 分钟.谁能告诉我如何增加更多分区以快速读取数据?

I am using spark 1.4 and I am trying to read the data from Hbase by using sc.newAPIHadoopRDD to read 2.7 GB data but there are 5 task are created for this stage and taking 2 t0 3 minutes to process it. Can anyone let me know how to increase the more partitions to read the data fast ?

推荐答案

org.apache.hadoop.hbase.mapreduce.TableInputFormat 为每个区域创建一个分区.您的表似乎分为 5 个区域.预先拆分你的表应该会增加分区的数量(看看 here 有关拆分的更多信息).

org.apache.hadoop.hbase.mapreduce.TableInputFormat creates a partition for each region. Your table seems to be split into 5 regions. Pre-splitting your table should increase the number of partitions (have a look here for more information on splitting).

这篇关于在 Spark sc.newAPIHadoopRDD 中读取 2.7 GB 数据，有 5 个分区的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！