使用 Spark 加载 CSV 文件 [英] Load CSV file with Spark

问题描述

我是 Spark 的新手，我正在尝试使用 Spark 从文件中读取 CSV 数据.这是我在做什么:

I'm new to Spark and I'm trying to read CSV data from a file with Spark. Here's what I am doing :

sc.textFile('file.csv')
    .map(lambda line: (line.split(',')[0], line.split(',')[1]))
    .collect()

我希望此调用能够为我提供文件前两列的列表，但出现此错误:

I would expect this call to give me a list of the two first columns of my file but I'm getting this error :

File "<ipython-input-60-73ea98550983>", line 1, in <lambda>
IndexError: list index out of range

虽然我的 CSV 文件不止一列.

although my CSV file as more than one column.

推荐答案

您确定所有行至少有 2 列吗?你可以尝试类似的东西来检查一下吗?:

Are you sure that all the lines have at least 2 columns? Can you try something like, just to check?:

sc.textFile("file.csv") \
    .map(lambda line: line.split(",")) \
    .filter(lambda line: len(line)>1) \
    .map(lambda line: (line[0],line[1])) \
    .collect()

或者，您可以打印罪魁祸首(如果有):

Alternatively, you could print the culprit (if any):

sc.textFile("file.csv") \
    .map(lambda line: line.split(",")) \
    .filter(lambda line: len(line)<=1) \
    .collect()

这篇关于使用 Spark 加载 CSV 文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！