使用Webhdfs/httpfs进行Spark [英] Spark with Webhdfs/httpfs

问题描述

我想通过httpfs或Webhdfs将HDFS中的文件读取到Spark中.类似于

I would like to read a file from HDFS into Spark via httpfs or Webhdfs. Something along the lines of

sc.textFile("webhdfs://myhost:14000/webhdfs/v1/path/to/file.txt")

sc.textFile("webhdfs://myhost:14000/webhdfs/v1/path/to/file.txt")

，或者理想情况下，

sc.textFile("httpfs://myhost:14000/webhdfs/v1/path/to/file.txt")

sc.textFile("httpfs://myhost:14000/webhdfs/v1/path/to/file.txt")

是否有办法让Spark通过Webhdfs/httpfs读取文件?

Is there a way to get Spark to read the file over Webhdfs/httpfs?

推荐答案

我相信WebHDFS/HttpFS就像流媒体源一样，可以通过REST-API传输数据.

I believe WebHDFS/ HttpFS are like streaming sources to transmit the data over REST-API.

然后，Spark Streaming可用于从WebHDFS/HttpFS接收数据.

Then Spark Streaming can be used to receive the data from the WebHDFS/ HttpFS.

这篇关于使用Webhdfs/httpfs进行Spark的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！