如何在不使用Scala案例类的情况下为CSV文件指定架构? [英] How to specify schema for CSV file without using Scala case class?

问题描述

我正在将CSV文件加载到DataFrame中，如下所示.

I am loading a CSV file into a DataFrame as below.

val conf=new SparkConf().setAppName("dataframes").setMaster("local")
val sc=new SparkContext(conf)
val spark=SparkSession.builder().getOrCreate()
import spark.implicits._

val df = spark.
  read.  
  format("org.apache.spark.csv").
  option("header", true).
  csv("/home/cloudera/Book1.csv")
scala> df.printSchema()
root
 |-- name: string (nullable = true)
 |-- address: string (nullable = true)
 |-- age: string (nullable = true)

如何将age列更改为Int类型?

推荐答案

有inferSchema选项可以通过以下方式自动识别变量的类型:

There is inferSchema option to automatically recognize the type of the variable by:

val df=spark.read
  .format("org.apache.spark.csv")
  .option("header", true)
  .option("inferSchema", true) // <-- HERE
  .csv("/home/cloudera/Book1.csv")

spark-csv最初是databricks的外部库，但从Spark 2.0版开始包含在核心spark中.您可以参考图书馆 github页面上的文档，以找到可用的选项.

spark-csv originally was an external library by databricks, but included in core spark from spark version 2.0 onwards. You can refer to documentation on the library's github page to find the available options.

这篇关于如何在不使用Scala案例类的情况下为CSV文件指定架构?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！