pyspark.sql.utils.IllegalArgumentException:“在Windows 10中实例化'org.apache.spark.sql.hive.HiveSessionStateBuild时出错 [英] pyspark.sql.utils.IllegalArgumentException: "Error while instantiating 'org.apache.spark.sql.hive.HiveSessionStateBuild in windows 10

问题描述

我已经在Windows 10中安装了带有winutils的spark 2.2.当我要运行pyspark时，我遇到了下面的异常

I have installed spark 2.2 with winutils in windows 10.when i am going to run pyspark i am facing bellow exception

pyspark.sql.utils.IllegalArgumentException: "Error while instantiating 'org.apache.spark.sql.hive.HiveSessionStateBuilder'

我已经在tmp/hive文件夹中尝试了权限777命令.但是暂时不起作用

I have already tried permission 777 commands in tmp/hive folder as well.but it is not working for now

winutils.exe chmod -R 777 C:\tmp\hive

应用此问题后，问题仍然相同.我在Windows 10中使用pyspark 2.2.她是无聊的环境

after applying this the problem remains same. I am using pyspark 2.2 in my windows 10. Her is spark-shell env

这里是pyspark shell

Here is pyspark shell

请帮助我弄清楚谢谢

推荐答案

端口9000 ?！它一定是与Hadoop相关的，因为我不记得Spark的端口.我建议您先使用 spark-shell 来消除任何其他跳数"，即 spark-shell 不需要两个运行时即可运行Spark本身和Python.

Port 9000?! It must be something Hadoop-related as I don't remember the port for Spark. I'd recommend using spark-shell first that would eliminate any additional "hops", i.e. spark-shell does not require two runtimes for Spark itself and Python.

考虑到例外情况，我非常确定问题是您有 some Hive-或与Hadoop相关的配置，Spark显然会使用它.

Given the exception I'm pretty sure that the issue is that you've got some Hive- or Hadoop-related configuration somewhere lying around and Spark uses it apparently.

原因"似乎表明在创建Spark SQL(即加载Hive的子系统)时使用了9000.

The "Caused by" seems to show that 9000 is used when Spark SQL is created which is when Hive-aware subsystem is loaded.

由以下原因引起:org.apache.spark.sql.AnalysisException:java.lang.RuntimeException:java.net.ConnectException:从DESKTOP-SDNSD47/192.168.10.143到0.0.0.0:9000的调用由于连接异常而失败:java.net.ConnectException:连接被拒绝

Caused by: org.apache.spark.sql.AnalysisException: java.lang.RuntimeException: java.net.ConnectException: Call From DESKTOP-SDNSD47/192.168.10.143 to 0.0.0.0:9000 failed on connection exception: java.net.ConnectException: Connection refused

请检查Windows 10中的环境变量(可能在命令行上使用 set 命令)，并删除所有与Hadoop相关的内容.

Please review the environment variables in Windows 10 (possibly using set command on command line) and remove anything Hadoop-related.

这篇关于pyspark.sql.utils.IllegalArgumentException:“在Windows 10中实例化'org.apache.spark.sql.hive.HiveSessionStateBuild时出错的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！