自定义定界符CSV阅读器Spark [英] Custom delimiter csv reader spark

问题描述

我想使用Apache Spark读取具有以下结构的文件.

I would like to read in a file with the following structure with Apache Spark.

628344092\t20070220\t200702\t2007\t2007.1370

定界符为\ t.在使用spark.read.csv()时如何实现呢?

The delimiter is \t. How can I implement this while using spark.read.csv()?

csv太大了，无法使用熊猫，因为它需要很长时间才能读取此文件.是否有某种类似于

The csv is much too big to use pandas because it takes ages to read this file. Is there some way which works similar to

pandas.read_csv(file, sep = '\t')

非常感谢！

推荐答案

使用spark.read.option("delimiter", "\t").csv(file)或sep代替delimiter.

如果字面上是\t，而不是制表符特殊字符，请使用双精度\:spark.read.option("delimiter", "\\t").csv(file)

If it's literally \t, not tab special character, use double \: spark.read.option("delimiter", "\\t").csv(file)

这篇关于自定义定界符CSV阅读器Spark的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！