使用Scala API将TSV读取到Spark Dataframe中 [英] Reading TSV into Spark Dataframe with Scala API

问题描述

我一直在尝试获取databricks库以读取CSV来工作.我正在尝试使用scala api将配置单元创建的TSV读取到spark数据框中.

I have been trying to get the databricks library for reading CSVs to work. I am trying to read a TSV created by hive into a spark data frame using the scala api.

这里有一个示例，您可以在spark shell中运行(我将示例数据公开，因此它可以为您工作)

Here is an example that you can run in the spark shell (I made the sample data public so it can work for you)

import org.apache.spark.sql.SQLContext
import org.apache.spark.sql.types.{StructType, StructField, StringType, IntegerType};

val sqlContext = new SQLContext(sc)
val segments = sqlContext.read.format("com.databricks.spark.csv").load("s3n://michaeldiscenza/data/test_segments")

文档说，您可以指定定界符，但我不清楚如何指定该选项

The documentation says you can specify the delimiter but I am unclear about how to specify that option.

推荐答案

所有选项参数都在option()函数中传递，如下所示:

All of the option parameters are passed in the option() function as below:

val segments = sqlContext.read.format("com.databricks.spark.csv")
    .option("delimiter", "\t")
    .load("s3n://michaeldiscenza/data/test_segments")

这篇关于使用Scala API将TSV读取到Spark Dataframe中的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！