如何在PySpark中使用read.csv跳过多行 [英] How to skip multiple lines using read.csv in PySpark

问题描述

我的.csv列很少，并且在使用spark.read.csv()函数将此文件导入数据帧时，希望跳过4(或通常为'n')行.我有这样的.csv文件-

I am having a .csv with few columns, and I wish to skip 4 (or 'n' in general) lines when importing this file into a dataframe using spark.read.csv() function. I have a .csv file like this -

ID;Name;Revenue
Identifier;Customer Name;Euros
cust_ID;cust_name;€
ID132;XYZ Ltd;2825
ID150;ABC Ltd;1849

在普通的Python中，使用read_csv()函数时，它很简单，并且可以使用skiprow=n选项(例如-

In normal Python, when using read_csv() function, it's simple and can be done using skiprow=n option like -

import pandas as pd
df=pd.read_csv('filename.csv',sep=';',skiprows=3) # Since we wish to skip top 3 lines

使用PySpark，我可以按如下方式导入此.csv文件-

With PySpark, I am importing this .csv file as follows -

df=spark.read.csv("filename.csv",sep=';') 
This imports the file as -
ID          |Name         |Revenue
Identifier  |Customer Name|Euros
cust_ID     |cust_name    |€
ID132       |XYZ Ltd      |2825
ID150       |ABC Ltd      1849

这是不正确的，因为我希望忽略前三行.我不能使用选项'header=True'，因为它只会排除第一行.可以使用'comment='选项，但为此需要以特殊字符开头的行，而我的文件则不是这样.我在文档中找不到任何内容.有什么办法可以实现?

This is not correct, because I wish to ignore first three lines. I can't use option 'header=True' because it will only exclude the first line. One can use 'comment=' option, but for that one needs the lines to start with a particular character and that is not the case with my file. I could not find anything in the documentation. Is there any way this can be accomplished?

推荐答案

我找不到适合您问题的简单解决方案.尽管无论标头如何写入都可以这样做，

I couldnt find a simple solution for your problem. Although this will work no matter how the header is written,

df = spark.read.csv("filename.csv",sep=';')\
          .rdd.zipWithIndex()\
          .filter(lambda x: x[1] > n)\
          .map(lambda x: x[0]).toDF()

这篇关于如何在PySpark中使用read.csv跳过多行的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！