如何在pyspark mapPartitions功能工作？ [英] How does the pyspark mapPartitions function work?

问题描述

所以我想学习使用Python（Pyspark）的火花。我想知道如何在功能 mapPartitions 的工作。这就是输入需要它给什么输出。我无法从网上找到任何合适的例子。比方说，我有一个包含列表，如低于RDD对象。

So I am trying to learn Spark using Python (Pyspark). I want to know how the function mapPartitions work. That is what Input it takes and what Output it gives. I couldn't find any proper example from the internet. Lets say, I have an RDD object containing lists, such as below.

[ [1, 2, 3], [3, 2, 4], [5, 2, 7] ] 

和我想从所有列表中删除元素2，我怎么会实现，使用 mapPartitions 。

And I want to remove element 2 from all the lists, how would I achieve that using mapPartitions.

推荐答案

mapPartition应该被认为是在分区映射操作，而不是在分区的元素。它的输入是一组当前分区的输出将是另一组分区。

mapPartition should be thought of as a map operation over Partitions and not over the elements of the partition. It's input is the set of current Partitions its output will be another set of Partitions.

您传递的地图必须将RDD的单个元素的功能

The function you pass map must take an individual element of your RDD

您传递mapPartition必须将RDD类型的迭代，并返回等一些或同类型的可迭代的funtion。

The funtion you pass mapPartition must take an iterable of your RDD type and return and iterable of some other or the same type.

在你的情况，你可能只想做这样的事情。

In your case you probably just want to do something like

def filterOut2(line):
    return [x for x in line if x != 2]

filtered_lists = data.map(filterOut2)

如果你想使用mapPartition这将是

if you wanted to use mapPartition it would be

def filterOut2FromPartion(list_of_lists):
  final_iterator = []
  for sub_list in list_of_lists:
    final_iterator.append( [x for x in sub_list if x != 2])
  return iter(final_iterator)

filtered_lists = data.mapPartition(filterOut2FromPartion)

这篇关于如何在pyspark mapPartitions功能工作？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！