sparklyr-连接远程hadoop集群 [英] sparklyr - Connect remote hadoop cluster

问题描述

是否可以将sparklyr与远程hadoop集群连接，或者只能在本地使用它?如果可能的话，怎么办?:)

It is possible to connect sparklyr with a remote hadoop cluster or it is only possible to use it local? And if it is possible, how? :)

在我看来，R通过火花到hadoop的连接非常重要！

In my opinion the connection from R to hadoop via spark is very important!

推荐答案

您可以使用 livy 这是用于Spark集群的Rest API服务.

You could use livy which is a Rest API service for the spark cluster.

在Azure上设置 HDinsight 群集后使用curl检查是否有常驻服务

once you have set up your HDinsight cluster on Azure check for livy service using curl

#curl test
curl -k --user "admin:mypassword1!" -v -X GET 


#r-studio code
sc <- spark_connect(master = "https://<yourclustername>.azurehdinsight.net/livy/",
                     method = "livy", config = livy_config(
                       username = "admin",
                       password = rstudioapi::askForPassword("Livy password:")))

一些有用的网址 https://docs.microsoft.com/en-us/azure/hdinsight/spark/apache-spark-livy-rest-interface

这篇关于sparklyr-连接远程hadoop集群的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！