在Apache Spark中读取包含分隔符的CSV文件 [英] Read CSV file with values containing delimiter in Apache spark

问题描述

读取 csv 文件(其中值包含 定界符)的有效方式是什么本身会出现 apache spark ?

what would be an efficient way to read a csv file in which the values are containing the delimiter itself in apache spark?

以下是我的数据集:

ID,Name,Age,Add,ress,Salary
1,Ross,32,Ah,med,abad,2000
2,Rachel,25,Delhi,1500
3,Chandler,23,Kota,2000
4,Monika,25,Mumbai,6500
5,Mike,27,Bhopal,8500
6,Phoebe,22,MP,4500
7,Joey,24,Indore,10000

推荐答案

{

//  1. read csv:
  val df1 = spark.read.option("header", "true").csv(fileFullName)
  df1.show(false)
// when you have format: 
//  ID,Name,Age,Add,ress,Salary
//  1,Ross,32,Ah,"med,abad",2000
//  2,Rachel,25,Delhi,,1500
//  3,Chandler,23,Kota,,2000
//  4,Monika,25,Mumbai,,6500
//  5,Mike,27,Bhopal,,8500
//  6,Phoebe,22,MP,,4500
//  7,Joey,24,Indore,,10000

//  3. result 


//    +---+--------+---+------+--------+------+
//    |ID |Name    |Age|Add   |ress    |Salary|
//    +---+--------+---+------+--------+------+
//    |1  |Ross    |32 |Ah    |med,abad|2000  |
//    |2  |Rachel  |25 |Delhi |null    |1500  |
//    |3  |Chandler|23 |Kota  |null    |2000  |
//    |4  |Monika  |25 |Mumbai|null    |6500  |
//    |5  |Mike    |27 |Bhopal|null    |8500  |
//    |6  |Phoebe  |22 |MP    |null    |4500  |
//    |7  |Joey    |24 |Indore|null    |10000 |
//    +---+--------+---+------+--------+------+

}

这篇关于在Apache Spark中读取包含分隔符的CSV文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！