火花运行与Java 8不工作 [英] Running yarn with spark not working with Java 8

问题描述

我有1个主站和6个从站，它们使用预编译版本的hadoop 2.6.0和spark 1.6.2。我正在运行hadoop MR和spark作业，而没有在所有节点上安装openjdk 7时出现任何问题。然而，当我在所有节点上将openjdk 7升级到openjdk 8时，spark会导致提交和spark-shell导致错误。

  16 / 08/17 14:06:22错误client.TransportClient：无法将RPC 4688442384427245199发送到/xxx.xxx.xxx.xx:42955：java.nio.channels.ClosedChannelExce ption 
 java.nio.channels.ClosedChannelException 
 16/08/17 14:06:22 WARN netty.NettyRpcEndpointRef：1次尝试
发送消息[message = RequestExecutors（0,0，Map（））]时出错org.apache.spark.SparkException： awaitResult中引发的异常$ or $ $ b $ org.apache.spark.rpc.RpcTimeout $$ anonfun $ 1.applyOrElse（RpcTimeout.scala：77）
 at org.apache.spark.rpc.RpcTimeout $$ anonfun $ 1。 applyOrElse（RpcTimeout.scala：75）
 at scala.runtime.AbstractPartialFunction.apply（AbstractPartialFunction.scala：36）
 at org.apache.spark.rpc.RpcTimeout $$ anonfun $ addMessageIfTimeout $ 1.applyOrElse（ RpcTimeout.scala：59）
在org.apache.spark.rpc.RpcTimeout $$ anonfun $ addMe ssageIfTimeout $ 1.applyOrElse（RpcTimeout.scala：59）
 at scala.PartialFunction $ OrElse.apply（PartialFunction.scala：167）
 at org.apache.spark.rpc.RpcTimeout.awaitResult（RpcTimeout.scala ：83）
 at org.apache.spark.rpc.RpcEndpointRef.askWithRetry（RpcEndpointRef.scala：102）
 at org.apache.spark.rpc.RpcEndpointRef.askWithRetry（RpcEndpointRef.scala：78）
 at org.apache.spark.scheduler.cluster.YarnSchedulerBackend $ YarnSchedulerEndpoint $$ anonfun $ receiveAndReply $ 1 $$ anonfun $ applyOrElse $ 1.apply $ m cV $ sp（YarnSchedulerBackend.scala：271）
 at org。 apache.spark.scheduler.cluster.YarnSchedulerBackend $ YarnSchedulerEndpoint $$ anonfun $ receiveAndReply $ 1 $$ anonfun $ applyOrElse $ 1.apply（Y arnSchedulerBackend.scala：271）
 at org.apache.spark.scheduler.cluster.YarnSchedulerBackend $ YarnSchedulerEndpoint $$ anonfun $ receiveAndReply $ 1 $$ anonfun $ applyOrElse $ 1.apply（Y arnSchedulerBackend.scala：271）
 at scala.concurrent.impl.F uture $ PromiseCompletingRunnable.liftedTree1 $ 1（Future.scala：24）
 at scala.concurrent.impl.Future $ PromiseCompletingRunnable.run（Future.scala：24）
 at java.util.concurrent.ThreadPoolExecutor.runWorker （ThreadPoolExecutor.java:1142）
在java.util.concurrent.ThreadPoolExecutor $ Worker.run（ThreadPoolExecutor.java:617）$ b $在java.lang.Thread.run（Thread.java:745）
导致：java.io.IOException：无法发送RPC 4688442384427245199到/xxx.xxx.xxx.xx:42955：java.nio.channels.ClosedChannelException $ b $在org.apache.spark.network.client .TransportClient $ 3.operationComplete（TransportClient.java:239）
 at org.apache.spark.network.client.TransportClient $ 3.operationComplete（TransportClient.java:226）
 at io.netty.util.concurrent .DefaultPromise.notifyListener0（DefaultPromise.java:680）
 at io.netty.util.concurrent.DefaultPromise $ LateListeners.run（DefaultPromise.java:845）
 at io.netty.util.concurrent .DefaultPromise $ LateListenerNotifier.run（DefaultPromise.java:873）
 at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTask​​s（SingleThreadEventExecutor.java:357）
 at io.netty.channel.nio.NioEventLoop .run（NioEventLoop.java:357）
 at io.netty.util.concurrent.SingleThreadEventExecutor $ 2.run（SingleThreadEventExecutor.java:111）
 ... 1 more 
原因：java .nio.channels.ClosedChannelException 
 16/08/17 14:06:22错误spark.SparkContext：初始化SparkContext时出错。 
 java.lang.IllegalStateException：在等待后端时停止了Spark上下文
在org.apache.spark.scheduler.TaskSchedulerImpl.waitBackendReady（TaskSchedulerImpl.scala：581）
在org.apache.spark .scheduler.TaskSchedulerImpl.postStartHook（TaskSchedulerImpl.scala：162）
 at org.apache.spark.SparkContext。< init>（SparkContext.scala：549）
 at org.apache.spark.api。在Sun.reflect.NativeConstructorAccessorImpl.newInstance0（本地方法）
（在Sun.reflect.NativeConstructorAccessorImpl.newInstance（NativeConstructorAccessorImpl.java:62）中）
（JavaSparkContext.scala：58） 
 at sun.reflect.DelegatingConstructorAccessorImpl.newInstance（DelegatingConstructorAccessorImpl.java:45）
 at java.lang.reflect.Constructor.newInstance（Constructor.java:423）
 at py4j.reflection.MethodInvoker .invoke（MethodInvoker.java:240）
 at py4j.reflection.ReflectionEngine.inv oke（ReflectionEngine.java:357）
 at py4j.Gateway.invoke（Gateway.java:236）
 at py4j.commands.ConstructorCommand.invokeConstructor（ConstructorCommand.java:80）
 at py4j .commands.ConstructorCommand.execute（ConstructorCommand.java:69）
 at py4j.GatewayConnection.run（GatewayConnection.java:211）
 at java.lang.Thread.run（Thread.java:745）
 Traceback（最近一次调用最后）：
在< module>文件中的/home/hd_spark/spark2/python/pyspark/shell.py，第49行， 
 spark = SparkSession.builder.getOrCreate（）
在getOrCreate 
 sc = SparkContext中的文件/home/hd_spark/spark2/python/pyspark/sql/session.py，第169行。 getOrCreate（sparkConf）
文件/home/hd_spark/spark2/python/pyspark/context.py，第294行，在getOrCreate 
 SparkContext（conf = conf或SparkConf（））
文件/home/hd_spark/spark2/python/pyspark/context.py，第115行，在__init__ 
 conf，jsc，profiler_cls）
文件/ home / hd_spark / spark2 / python / pyspark / context .py，第168行，在_do_init 
 self._jsc = jsc或self._initialize_context（self._conf._jconf）
文件/home/hd_spark/spark2/python/pyspark/context.py ，第233行，在_initialize_context 
中返回self._jvm.JavaSparkContext（jconf）
文件/home/hd_spark/spark2/python/lib/py4j-0.10.1-src.zip/py4j/java_gateway。 py，行1183，在__call__ 
文件/home/hd_spark/spark2/python/lib/py4j-0.10.1-src.zip/py4j/protocol.py，第312行，在get_return_value 
 py4j.protocol.Py 4JJavaError：调用None.org.apache.spark.api.java.JavaSparkContext时发生错误。 
：java.lang.IllegalStateException：在等待后端时停止Spark上下文
在org.apache.spark.scheduler.TaskSchedulerImpl.waitBackendReady（TaskSchedulerImpl.scala：581）
在org.apache。 spark.scheduler.TaskSchedulerImpl.postStartHook（TaskSchedulerImpl.scala：162）
 at org.apache.spark.SparkContext。< init>（SparkContext.scala：549）
 at org.apache.spark.api .java.JavaSparkContext。< init>（JavaSparkContext.scala：58）
 at sun.reflect.NativeConstructorAccessorImpl.newInstance0（Native Method）
 at sun.reflect.NativeConstructorAccessorImpl.newInstance（NativeConstructorAccessorImpl.java:62 ）
 at sun.reflect.DelegatingConstructorAccessorImpl.newInstance（DelegatingConstructorAccessorImpl.java:45）
 at java.lang.reflect.Constructor.newInstance（Constructor.java:423）
 at py4j.reflection。 MethodInvoker.invoke（MethodInvoker.java:240）
 at py4j.reflection.ReflectionEngine.i在py4j.Gateway.invoke（Gateway.java:236）
（py4j.commands.ConstructorCommand.invokeConstructor（ConstructorCommand.java:80）
在py4j 
 .commands.ConstructorCommand.execute（ConstructorCommand.java:69）
 at py4j.GatewayConnection.run（GatewayConnection.java:211）
 at java.lang.Thread.run（Thread.java:745）
  

我在.bashrc上导出了JAVA_HOME，并使用

$将openjdk 8设置为默认java b
$ b pre $ sudo update-alternatives --config java
sudo update-alternatives --config javac

这些命令。此外，我已经尝试与甲骨文Java 8和相同的错误出现。

  SLF4J：类路径包含多个SLF4J绑定。 
 SLF4J：在[jar：file：/ tmp / hadoop-hd_spark / nm-local-dir / usercache / hd_spark / filecache / 17 / __ spark_libs__8247267244939901627.zip/slf4j-log4j12-1.7.16.jar!/中找到绑定org / slf4j / impl / StaticLoggerBinder.class] 
 SLF4J：在[jar：file：/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar！/ org / slf4j / impl / StaticLoggerBinder.class] 
 SLF4J：有关说明，请参阅http://www.slf4j.org/codes.html#multiple_bindings。 
 SLF4J：实际绑定类型为[org.slf4j.impl.Log4jLoggerFactory] ​​
 16/08/17 14:05:11信息executor.CoarseGrainedExecutorBackend：已启动守护进程，进程名称为：23541 @ slave01 
 16/08/17 14:05:11 INFO util.SignalUtils：TERM的注册信号处理程序
 16/08/17 14:05:11 INFO util.SignalUtils：HUP 
的注册信号处理程序16/08/17 14:05:11 INFO util.SignalUtils：INT 
的注册信号处理程序16/08/17 14:05:11 WARN util.NativeCodeLoader：无法为您的平台加载native-hadoop库。 ..在适用的地方使用builtin-java类
 16/08/17 14:05:11 INFO spark.SecurityManager：将视图acls更改为：hd_spark 
 16/08/17 14:05:11 INFO spark .SecurityManager：将修改的acls更改为：hd_spark 
 16/08/17 14:05:11 INFO spark.SecurityManager：将视图acls组更改为：
 16/08/17 14:05:11 INFO spark .SecurityManager：将修改的acls组更改为：
 16/08/17 14:05:11 INFO spark.SecurityManager：SecurityManager：身份验证二sabled;用户禁用;拥有查看权限的用户：Set（hd_spark）;具有查看权限的组：Set（）;具有修改权限的用户：Set（hd_spark）;具有修改权限的组：Set（）
 16/08/17 14:05:12 INFO client.TransportClientFactory：在78 ms后成功创建了到/xxx.xxx.xxx.xx:37417的连接（0 ms用于引导）
 16/08/17 14:05:12 INFO spark.SecurityManager：将视图acls更改为：hd_spark 
 16/08/17 14:05:12 INFO spark.SecurityManager：将修改的acls更改为： hd_spark 
 16/08/17 14:05:12 INFO spark.SecurityManager：将视图的acls组更改为：
 16/08/17 14:05:12 INFO spark.SecurityManager：将修改的acls组更改为：
 16/08/17 14:05:12 INFO spark.SecurityManager：SecurityManager：身份验证已禁用;用户禁用;拥有查看权限的用户：Set（hd_spark）;具有查看权限的组：Set（）;具有修改权限的用户：Set（hd_spark）;具有修改权限的组：Set（）
 16/08/17 14:05:12 INFO client.TransportClientFactory：在1 ms后成功创建了到/xxx.xxx.xxx.xx:37417的连接（0 ms用于引导）
 16/08/17 14:05:12 INFO storage.DiskBlockManager：在/ tmp / hadoop-hd_spark / nm-local-dir / usercache / hd_spark / appcache / application_1471352972661_0005 / blockmgr-d9f23a56-1420创建本地目录-4cd4-abfd-ae9e128c688c 
 16/08/17 14:05:12 INFO memory.MemoryStore：MemoryStore以容量开始366.3 MB 
 16/08/17 14:05:12信息executor.CoarseGrainedExecutorBackend：连接到驱动程序：spark：//[e-mail]：37417 
 16/08/17 14:05:13 ERROR executor.CoarseGrainedExecutorBackend：收到的信号期限
 16/08/17 14:05:13 INFO storage.DiskBlockManager：Shutdown hook called 
 16/08/17 14:05:13 INFO util.ShutdownHookManager：Shutdown hook called 
  

我已经尝试过使用spark 1.6.2预建版本，spark 2.0预建版本和al所以试着用spark 2.0自己构建它。

即使在升级到java 8之后，Hadoop作业仍然完美。当我切换回到java 7时，spark工作正常。

我的scala版本是2.11，操作系统是Ubuntu 14.04.4 LTS。

如果有人可以给我一个想法来解决这个问题。

谢谢！

ps我在日志中将我的IP地址更改为xxx.xxx.xxx.xx。

解决方案

截至2016年9月12日，这是一个阻碍问题： https://issues.apache.org/jira/browse/YARN-4714

你可以通过在yarn- site.xml

 < property> 
< name> yarn.nodemanager.pmem-check-enabled< / name> 
<值> false< /值> 
< / property> 
 
<属性> 
< name> yarn.nodemanager.vmem-check-enabled< / name> 
<值> false< /值> 
< / property> 
  

I have cluster with 1 master and 6 slaves which uses pre-built version of hadoop 2.6.0 and spark 1.6.2. I was running hadoop MR and spark jobs without any problem with openjdk 7 installed on all the nodes. However when I upgraded openjdk 7 to openjdk 8 on all nodes, spark submit and spark-shell with yarn caused error.

16/08/17 14:06:22 ERROR client.TransportClient: Failed to send RPC 4688442384427245199 to /xxx.xxx.xxx.xx:42955: java.nio.channels.ClosedChannelExce      ption
java.nio.channels.ClosedChannelException
16/08/17 14:06:22 WARN netty.NettyRpcEndpointRef: Error sending message [message = RequestExecutors(0,0,Map())] in 1 attempts
org.apache.spark.SparkException: Exception thrown in awaitResult
        at org.apache.spark.rpc.RpcTimeout$$anonfun$1.applyOrElse(RpcTimeout.scala:77)
        at org.apache.spark.rpc.RpcTimeout$$anonfun$1.applyOrElse(RpcTimeout.scala:75)
        at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
        at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:59)
        at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:59)
        at scala.PartialFunction$OrElse.apply(PartialFunction.scala:167)
        at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:83)
        at org.apache.spark.rpc.RpcEndpointRef.askWithRetry(RpcEndpointRef.scala:102)
        at org.apache.spark.rpc.RpcEndpointRef.askWithRetry(RpcEndpointRef.scala:78)
        at org.apache.spark.scheduler.cluster.YarnSchedulerBackend$YarnSchedulerEndpoint$$anonfun$receiveAndReply$1$$anonfun$applyOrElse$1.apply$m      cV$sp(YarnSchedulerBackend.scala:271)
        at org.apache.spark.scheduler.cluster.YarnSchedulerBackend$YarnSchedulerEndpoint$$anonfun$receiveAndReply$1$$anonfun$applyOrElse$1.apply(Y      arnSchedulerBackend.scala:271)
        at org.apache.spark.scheduler.cluster.YarnSchedulerBackend$YarnSchedulerEndpoint$$anonfun$receiveAndReply$1$$anonfun$applyOrElse$1.apply(Y      arnSchedulerBackend.scala:271)
        at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
        at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Failed to send RPC 4688442384427245199 to /xxx.xxx.xxx.xx:42955: java.nio.channels.ClosedChannelException
        at org.apache.spark.network.client.TransportClient$3.operationComplete(TransportClient.java:239)
        at org.apache.spark.network.client.TransportClient$3.operationComplete(TransportClient.java:226)
        at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:680)
        at io.netty.util.concurrent.DefaultPromise$LateListeners.run(DefaultPromise.java:845)
        at io.netty.util.concurrent.DefaultPromise$LateListenerNotifier.run(DefaultPromise.java:873)
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:357)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357)
        at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
        ... 1 more
Caused by: java.nio.channels.ClosedChannelException
16/08/17 14:06:22 ERROR spark.SparkContext: Error initializing SparkContext.
java.lang.IllegalStateException: Spark context stopped while waiting for backend
        at org.apache.spark.scheduler.TaskSchedulerImpl.waitBackendReady(TaskSchedulerImpl.scala:581)
        at org.apache.spark.scheduler.TaskSchedulerImpl.postStartHook(TaskSchedulerImpl.scala:162)
        at org.apache.spark.SparkContext.<init>(SparkContext.scala:549)
        at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:240)
        at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
        at py4j.Gateway.invoke(Gateway.java:236)
        at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)
        at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69)
        at py4j.GatewayConnection.run(GatewayConnection.java:211)
        at java.lang.Thread.run(Thread.java:745)
Traceback (most recent call last):
  File "/home/hd_spark/spark2/python/pyspark/shell.py", line 49, in <module>
    spark = SparkSession.builder.getOrCreate()
  File "/home/hd_spark/spark2/python/pyspark/sql/session.py", line 169, in getOrCreate
    sc = SparkContext.getOrCreate(sparkConf)
  File "/home/hd_spark/spark2/python/pyspark/context.py", line 294, in getOrCreate
    SparkContext(conf=conf or SparkConf())
  File "/home/hd_spark/spark2/python/pyspark/context.py", line 115, in __init__
    conf, jsc, profiler_cls)
  File "/home/hd_spark/spark2/python/pyspark/context.py", line 168, in _do_init
    self._jsc = jsc or self._initialize_context(self._conf._jconf)
  File "/home/hd_spark/spark2/python/pyspark/context.py", line 233, in _initialize_context
    return self._jvm.JavaSparkContext(jconf)
  File "/home/hd_spark/spark2/python/lib/py4j-0.10.1-src.zip/py4j/java_gateway.py", line 1183, in __call__
  File "/home/hd_spark/spark2/python/lib/py4j-0.10.1-src.zip/py4j/protocol.py", line 312, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling None.org.apache.spark.api.java.JavaSparkContext.
: java.lang.IllegalStateException: Spark context stopped while waiting for backend
        at org.apache.spark.scheduler.TaskSchedulerImpl.waitBackendReady(TaskSchedulerImpl.scala:581)
        at org.apache.spark.scheduler.TaskSchedulerImpl.postStartHook(TaskSchedulerImpl.scala:162)
        at org.apache.spark.SparkContext.<init>(SparkContext.scala:549)
        at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:240)
        at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
        at py4j.Gateway.invoke(Gateway.java:236)
        at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)
        at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69)
        at py4j.GatewayConnection.run(GatewayConnection.java:211)
        at java.lang.Thread.run(Thread.java:745)

I have exported JAVA_HOME on .bashrc and have set the openjdk 8 as default java using

sudo update-alternatives --config java
sudo update-alternatives --config javac

these commands. Also I have tried with oracle java 8 and the same error comes up. The container logs on the slave nodes have same error as below.

SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/hadoop-hd_spark/nm-local-dir/usercache/hd_spark/filecache/17/__spark_libs__8247267244939901627.zip/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
16/08/17 14:05:11 INFO executor.CoarseGrainedExecutorBackend: Started daemon with process name: 23541@slave01
16/08/17 14:05:11 INFO util.SignalUtils: Registered signal handler for TERM
16/08/17 14:05:11 INFO util.SignalUtils: Registered signal handler for HUP
16/08/17 14:05:11 INFO util.SignalUtils: Registered signal handler for INT
16/08/17 14:05:11 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
16/08/17 14:05:11 INFO spark.SecurityManager: Changing view acls to: hd_spark
16/08/17 14:05:11 INFO spark.SecurityManager: Changing modify acls to: hd_spark
16/08/17 14:05:11 INFO spark.SecurityManager: Changing view acls groups to:
16/08/17 14:05:11 INFO spark.SecurityManager: Changing modify acls groups to:
16/08/17 14:05:11 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(hd_spark); groups with view permissions: Set(); users  with modify permissions: Set(hd_spark); groups with modify permissions: Set()
16/08/17 14:05:12 INFO client.TransportClientFactory: Successfully created connection to /xxx.xxx.xxx.xx:37417 after 78 ms (0 ms spent in bootstraps)
16/08/17 14:05:12 INFO spark.SecurityManager: Changing view acls to: hd_spark
16/08/17 14:05:12 INFO spark.SecurityManager: Changing modify acls to: hd_spark
16/08/17 14:05:12 INFO spark.SecurityManager: Changing view acls groups to:
16/08/17 14:05:12 INFO spark.SecurityManager: Changing modify acls groups to:
16/08/17 14:05:12 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(hd_spark); groups with view permissions: Set(); users  with modify permissions: Set(hd_spark); groups with modify permissions: Set()
16/08/17 14:05:12 INFO client.TransportClientFactory: Successfully created connection to /xxx.xxx.xxx.xx:37417 after 1 ms (0 ms spent in bootstraps)
16/08/17 14:05:12 INFO storage.DiskBlockManager: Created local directory at /tmp/hadoop-hd_spark/nm-local-dir/usercache/hd_spark/appcache/application_1471352972661_0005/blockmgr-d9f23a56-1420-4cd4-abfd-ae9e128c688c
16/08/17 14:05:12 INFO memory.MemoryStore: MemoryStore started with capacity 366.3 MB
16/08/17 14:05:12 INFO executor.CoarseGrainedExecutorBackend: Connecting to driver: spark://[e-mail]:37417
16/08/17 14:05:13 ERROR executor.CoarseGrainedExecutorBackend: RECEIVED SIGNAL TERM
16/08/17 14:05:13 INFO storage.DiskBlockManager: Shutdown hook called
16/08/17 14:05:13 INFO util.ShutdownHookManager: Shutdown hook called

I have tried with spark 1.6.2 pre-built version, spark 2.0 pre-built version and also tried with spark 2.0 by building it myself.

Hadoop job works perfectly even after upgrading to java 8. When i switch back to java 7, spark works fine.

My scala version is 2.11 and OS is Ubuntu 14.04.4 LTS .

It will be very great if someone can give me an idea to solve this problem.

Thanks!

ps I have changed my IP address as xxx.xxx.xxx.xx on the logs.

解决方案

As of September 12, 2016, this is a blocker issue: https://issues.apache.org/jira/browse/YARN-4714

You can overcome this by setting up the following properties in yarn-site.xml

<property>
    <name>yarn.nodemanager.pmem-check-enabled</name>
    <value>false</value>
</property>

<property>
    <name>yarn.nodemanager.vmem-check-enabled</name>
    <value>false</value>
</property>

这篇关于火花运行与Java 8不工作的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！