将REST API中的CSV处理到Spark中 [英] Process CSV from REST API into Spark

问题描述

从rest api直接将csv格式的结果直接读取到spark中的最佳方法是什么?

What is the best way to read a csv formatted result from a rest api directly into spark?

基本上有这个，我知道我可以在scala中处理并将其保存到文件，但想在spark中处理数据:

Basically have this which I know I can process in scala and save to a file but would like to process the data in spark:

val resultCsv = scala.io.Source.fromURL(url).getLines()

推荐答案

这是可以完成的方法.

对于Spark 2.2.x

import scala.io.Source._
import org.apache.spark.sql.{Dataset, SparkSession}

var res = fromURL(url).mkString.stripMargin.lines.toList
val csvData: Dataset[String] = spark.sparkContext.parallelize(res).toDS()

val frame = spark.read.option("header", true).option("inferSchema",true).csv(csvData)
frame.printSchema()

将databricks库用于旧版本的Spark

using databricks lib for older version of Spark

import scala.io.Source._
import com.databricks.spark.csv.CsvParser

var res = fromURL(url).mkString.stripMargin.lines.toList
val csvData: Dataset[String] = spark.sparkContext.parallelize(res)

val csvParser = new CsvParser()
  .withUseHeader(true)
  .withInferSchema(true)

val frame: DataFrame = csvParser.csvRdd(sqlContext, rdd)
frame.printSchema()

注意:-我是Scala的新手，任何改进将不胜感激.

Note:- I am new to Scala any improvements will be appreciated.

ref: 查看全文