如何使用Spark-Scala从Web下载CSV文件? [英] How to use Spark-Scala to download a CSV file from the web?
本文介绍了如何使用Spark-Scala从Web下载CSV文件?的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!
问题描述
世界
如何使用Spark-Scala从Web下载CSV文件并将其加载到spark-csv DataFrame中?
How to use Spark-Scala to download a CSV file from the web and load the file into a spark-csv DataFrame?
当前,我依靠shell命令中的curl来获取我的CSV文件.
Currently I depend on curl in a shell command to get my CSV file.
这是我要增强的语法:
/* fb_csv.scala
This script should load FB prices from Yahoo.
Demo:
spark-shell -i fb_csv.scala
*/
// I should get prices:
import sys.process._
"/usr/bin/curl -o /tmp/fb.csv http://ichart.finance.yahoo.com/table.csv?s=FB"!
import org.apache.spark.sql.SQLContext
val sqlContext = new SQLContext(sc)
val fb_df = sqlContext.read.format("com.databricks.spark.csv").option("header","true").option("inferSchema","true").load("/tmp/fb.csv")
fb_df.head(9)
我想增强上面的脚本,所以它是纯Scala,内部没有shell语法.
I want to enhance the above script so it is pure Scala with no shell syntax inside.
推荐答案
val content = scala.io.Source.fromURL("http://ichart.finance.yahoo.com/table.csv?s=FB").mkString
val list = content.split("\n").filter(_ != "")
val rdd = sc.parallelize(list)
val df = rdd.toDF
这篇关于如何使用Spark-Scala从Web下载CSV文件?的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!
查看全文