断开连接后，Spark无法在本地模式下启动[Spark中处理IPv6的可能错误?] [英] Spark fails to start in local mode when disconnected [Possible bug in handling IPv6 in Spark??]
本文介绍了断开连接后，Spark无法在本地模式下启动[Spark中处理IPv6的可能错误?]的处理方法，对大家解决问题具有一定的参考价值，需要的朋友们下面随着小编来一起学习吧！
问题描述
问题与此处所述相同启动spark-在Mac上运行Shell本地
...但是我没有找到解决方案.我还曾经收到格式错误的URI错误，但现在我得到了预期的主机名.
... but I have failed to find a solution. I also used to get the malformed URI error but now I get expected hostname.
因此，当我未连接到Internet时，Spark Shell无法在本地模式下加载[请参见下面的错误].因此，我正在运行从Internet下载的Apache Spark 2.1.0，并在我的Mac上运行.所以我运行./bin/spark-shell，它给了我下面的错误.
So when I am not connected to internet, spark shell fails to load in local mode [See the error below]. So I am running Apache Spark 2.1.0 downloaded from internet, running on my Mac. So I run ./bin/spark-shell and it gives me the error below.
所以我已经阅读了Spark代码，它正在使用Java的InetAddress.getLocalHost()查找本地主机的IP地址.因此，当我连接到Internet时，我将使用本地主机名获取一个IPv4.
So I have read the Spark code and it is using Java's InetAddress.getLocalHost() to find the localhost's IP address. So when I am connected to internet, I get back an IPv4 with my local hostname.
scala> InetAddress.getLocalHost
res9: java.net.InetAddress = AliKheyrollahis-MacBook-Pro.local/192.168.1.26
但是密钥是，当断开连接时，我得到一个IPv6，其值的百分比(范围内):
but the key is, when disconnected, I get an IPv6 with a percentage in the values (it is scoped):
scala> InetAddress.getLocalHost
res10: java.net.InetAddress = AliKheyrollahis-MacBook-Pro.local/fe80:0:0:0:2b9a:4521:a301:e9a5%10
此IP与您在错误消息中看到的IP相同.我觉得我的问题是，由于无法处理结果中的%10，因此会抛出Spark.
And this IP is the same as the one you see in the error message. I feel my problem is that it throws Spark since it cannot handle %10 in the result.
我的猜测是这是个错误，很可能很少有人看到，因为人们总是连接到Internet或他们的Mac不会返回范围限定的IPv6.即使可以配置Mac来解决此问题，我也很高兴.我做了任何事情，包括将IPv6设置为手动或仅本地链接无效.
My guess is this is a bug, probably witnessed by very few since people always connected to internet or their mac does not return a scoped IPv6. Even if I can configure my Mac to get around this issue, I am happy. I have done anything including setting IPv6 to manual or Link-local only to no avail.
我还尝试了删除/etc/hosts中的::1 localhost行，但无济于事.
I have also tried removing ::1 localhost line in /etc/hosts to no avail.
这是DEBUG输出的完整错误(请注意，用于侦听的是相同的IPv6):
So here is the full error with DEBUG output (please note the same IPv6 being used to listen):
7/01/28 22:02:59 DEBUG ShutdownHookManager: Adding shutdown hook
17/01/28 22:03:06 DEBUG Shell: setsid is not available on this machine. So not using it.
17/01/28 22:03:06 DEBUG Shell: setsid exited with exit code 0
17/01/28 22:03:06 INFO SparkContext: Running Spark version 2.1.0
17/01/28 22:03:06 DEBUG MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginSuccess with annotation @org.apache.hadoop.metrics2.annotation.Metric(sampleName=Ops, always=false, about=, type=DEFAULT, value=[Rate of successful kerberos logins and latency (milliseconds)], valueName=Time)
17/01/28 22:03:06 DEBUG MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginFailure with annotation @org.apache.hadoop.metrics2.annotation.Metric(sampleName=Ops, always=false, about=, type=DEFAULT, value=[Rate of failed kerberos logins and latency (milliseconds)], valueName=Time)
17/01/28 22:03:06 DEBUG MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.getGroups with annotation @org.apache.hadoop.metrics2.annotation.Metric(sampleName=Ops, always=false, about=, type=DEFAULT, value=[GetGroups], valueName=Time)
17/01/28 22:03:06 DEBUG MetricsSystemImpl: UgiMetrics, User and group related metrics
17/01/28 22:03:26 DEBUG KerberosName: Kerberos krb5 configuration not found, setting default realm to empty
17/01/28 22:03:26 DEBUG Groups: Creating new Groups object
17/01/28 22:03:26 DEBUG NativeCodeLoader: Trying to load the custom-built native-hadoop library...
17/01/28 22:03:26 DEBUG NativeCodeLoader: Failed to load native-hadoop with error: java.lang.UnsatisfiedLinkError: no hadoop in java.library.path
17/01/28 22:03:26 DEBUG NativeCodeLoader: java.library.path=/Users/aliostad/torch/install/lib:/Users/aliostad/torch/install/lib:/Users/aliostad/torch/install/lib:/Users/aliostad/torch/install/lib:/Users/aliostad/torch/install/lib:/Users/aliostad/torch/install/lib:/Users/aliostad/torch/install/lib::/Users/aliostad/Library/Java/Extensions:/Library/Java/Extensions:/Network/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java:.
17/01/28 22:03:26 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17/01/28 22:03:26 DEBUG PerformanceAdvisory: Falling back to shell based
17/01/28 22:03:26 DEBUG JniBasedUnixGroupsMappingWithFallback: Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping
17/01/28 22:03:27 DEBUG Groups: Group mapping impl=org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback; cacheTimeout=300000; warningDeltaMs=5000
17/01/28 22:03:27 DEBUG UserGroupInformation: hadoop login
17/01/28 22:03:27 DEBUG UserGroupInformation: hadoop login commit
17/01/28 22:03:27 DEBUG UserGroupInformation: using local user:UnixPrincipal: aliostad
17/01/28 22:03:27 DEBUG UserGroupInformation: Using user: "UnixPrincipal: aliostad" with name aliostad
17/01/28 22:03:27 DEBUG UserGroupInformation: User entry: "aliostad"
17/01/28 22:03:27 DEBUG UserGroupInformation: UGI loginUser:aliostad (auth:SIMPLE)
17/01/28 22:03:27 INFO SecurityManager: Changing view acls to: aliostad
17/01/28 22:03:27 INFO SecurityManager: Changing modify acls to: aliostad
17/01/28 22:03:27 INFO SecurityManager: Changing view acls groups to:
17/01/28 22:03:27 INFO SecurityManager: Changing modify acls groups to:
17/01/28 22:03:27 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(aliostad); groups with view permissions: Set(); users with modify permissions: Set(aliostad); groups with modify permissions: Set()
17/01/28 22:03:27 DEBUG SecurityManager: Created SSL options for fs: SSLOptions{enabled=false, keyStore=None, keyStorePassword=None, trustStore=None, trustStorePassword=None, protocol=None, enabledAlgorithms=Set()}
17/01/28 22:03:27 DEBUG InternalLoggerFactory: Using SLF4J as the default logging framework
17/01/28 22:03:27 DEBUG PlatformDependent0: java.nio.Buffer.address: available
17/01/28 22:03:27 DEBUG PlatformDependent0: sun.misc.Unsafe.theUnsafe: available
17/01/28 22:03:27 DEBUG PlatformDependent0: sun.misc.Unsafe.copyMemory: available
17/01/28 22:03:27 DEBUG PlatformDependent0: direct buffer constructor: available
17/01/28 22:03:27 DEBUG PlatformDependent0: java.nio.Bits.unaligned: available, true
17/01/28 22:03:27 DEBUG PlatformDependent0: java.nio.DirectByteBuffer.<init>(long, int): available
17/01/28 22:03:27 DEBUG Cleaner0: java.nio.ByteBuffer.cleaner(): available
17/01/28 22:03:27 DEBUG PlatformDependent: Java version: 8
17/01/28 22:03:27 DEBUG PlatformDependent: -Dio.netty.noUnsafe: false
17/01/28 22:03:27 DEBUG PlatformDependent: sun.misc.Unsafe: available
17/01/28 22:03:27 DEBUG PlatformDependent: -Dio.netty.noJavassist: false
17/01/28 22:03:27 DEBUG PlatformDependent: Javassist: available
17/01/28 22:03:27 DEBUG PlatformDependent: -Dio.netty.tmpdir: /var/folders/pz/vgqg2gns18j_kxsnkzrp6x_m0000gn/T (java.io.tmpdir)
17/01/28 22:03:27 DEBUG PlatformDependent: -Dio.netty.bitMode: 64 (sun.arch.data.model)
17/01/28 22:03:27 DEBUG PlatformDependent: -Dio.netty.noPreferDirect: false
17/01/28 22:03:27 DEBUG PlatformDependent: io.netty.maxDirectMemory: 0 bytes
17/01/28 22:03:27 DEBUG JavassistTypeParameterMatcherGenerator: Generated: io.netty.util.internal.__matchers__.org.apache.spark.network.protocol.MessageMatcher
17/01/28 22:03:27 DEBUG JavassistTypeParameterMatcherGenerator: Generated: io.netty.util.internal.__matchers__.io.netty.buffer.ByteBufMatcher
17/01/28 22:03:27 DEBUG MultithreadEventLoopGroup: -Dio.netty.eventLoopThreads: 8
17/01/28 22:03:27 DEBUG NioEventLoop: -Dio.netty.noKeySetOptimization: false
17/01/28 22:03:27 DEBUG NioEventLoop: -Dio.netty.selectorAutoRebuildThreshold: 512
17/01/28 22:03:27 DEBUG PlatformDependent: org.jctools-core.MpscChunkedArrayQueue: available
17/01/28 22:03:27 DEBUG PooledByteBufAllocator: -Dio.netty.allocator.numHeapArenas: 8
17/01/28 22:03:27 DEBUG PooledByteBufAllocator: -Dio.netty.allocator.numDirectArenas: 8
17/01/28 22:03:27 DEBUG PooledByteBufAllocator: -Dio.netty.allocator.pageSize: 8192
17/01/28 22:03:27 DEBUG PooledByteBufAllocator: -Dio.netty.allocator.maxOrder: 11
17/01/28 22:03:27 DEBUG PooledByteBufAllocator: -Dio.netty.allocator.chunkSize: 16777216
17/01/28 22:03:27 DEBUG PooledByteBufAllocator: -Dio.netty.allocator.tinyCacheSize: 512
17/01/28 22:03:27 DEBUG PooledByteBufAllocator: -Dio.netty.allocator.smallCacheSize: 256
17/01/28 22:03:27 DEBUG PooledByteBufAllocator: -Dio.netty.allocator.normalCacheSize: 64
17/01/28 22:03:27 DEBUG PooledByteBufAllocator: -Dio.netty.allocator.maxCachedBufferCapacity: 32768
17/01/28 22:03:27 DEBUG PooledByteBufAllocator: -Dio.netty.allocator.cacheTrimInterval: 8192
17/01/28 22:03:27 DEBUG ThreadLocalRandom: -Dio.netty.initialSeedUniquifier: 0x3185a000d3a47bd4 (took 1 ms)
17/01/28 22:03:27 DEBUG ByteBufUtil: -Dio.netty.allocator.type: unpooled
17/01/28 22:03:27 DEBUG ByteBufUtil: -Dio.netty.threadLocalDirectBufferSize: 65536
17/01/28 22:03:27 DEBUG ByteBufUtil: -Dio.netty.maxThreadLocalCharBufferSize: 16384
17/01/28 22:03:27 DEBUG NetUtil: Loopback interface: lo0 (lo0, 0:0:0:0:0:0:0:1)
17/01/28 22:03:27 DEBUG NetUtil: /proc/sys/net/core/somaxconn: 128 (non-existent)
17/01/28 22:03:27 DEBUG TransportServer: Shuffle server started on port: 56107
17/01/28 22:03:27 INFO Utils: Successfully started service 'sparkDriver' on port 56107.
17/01/28 22:03:27 DEBUG SparkEnv: Using serializer: class org.apache.spark.serializer.JavaSerializer
17/01/28 22:03:27 INFO SparkEnv: Registering MapOutputTracker
17/01/28 22:03:27 DEBUG MapOutputTrackerMasterEndpoint: init
17/01/28 22:03:27 INFO SparkEnv: Registering BlockManagerMaster
17/01/28 22:03:27 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
17/01/28 22:03:27 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
17/01/28 22:03:28 INFO DiskBlockManager: Created local directory at /private/var/folders/pz/vgqg2gns18j_kxsnkzrp6x_m0000gn/T/blockmgr-4079e45b-e4e0-4386-bffe-42af18634710
17/01/28 22:03:28 DEBUG DiskBlockManager: Adding shutdown hook
17/01/28 22:03:28 INFO MemoryStore: MemoryStore started with capacity 366.3 MB
17/01/28 22:03:28 INFO SparkEnv: Registering OutputCommitCoordinator
17/01/28 22:03:28 DEBUG OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: init
17/01/28 22:03:28 DEBUG SecurityManager: Created SSL options for ui: SSLOptions{enabled=false, keyStore=None, keyStorePassword=None, trustStore=None, trustStorePassword=None, protocol=None, enabledAlgorithms=Set()}
17/01/28 22:03:28 INFO Utils: Successfully started service 'SparkUI' on port 4040.
17/01/28 22:03:28 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://fe80:0:0:0:2b9a:4521:a301:e9a5%10:4040
17/01/28 22:03:28 INFO Executor: Starting executor ID driver on host localhost
17/01/28 22:03:28 INFO Executor: Using REPL class URI: spark://fe80:0:0:0:2b9a:4521:a301:e9a5%10:56107/classes
17/01/28 22:03:28 ERROR SparkContext: Error initializing SparkContext.
java.lang.AssertionError: assertion failed: Expected hostname
at scala.Predef$.assert(Predef.scala:170)
at org.apache.spark.util.Utils$.checkHost(Utils.scala:931)
at org.apache.spark.util.RpcUtils$.makeDriverRef(RpcUtils.scala:31)
at org.apache.spark.executor.Executor.<init>(Executor.scala:121)
at org.apache.spark.scheduler.local.LocalEndpoint.<init>(LocalSchedulerBackend.scala:59)
at org.apache.spark.scheduler.local.LocalSchedulerBackend.start(LocalSchedulerBackend.scala:126)
at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:156)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:509)
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2313)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:868)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:860)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:860)
at org.apache.spark.repl.Main$.createSparkSession(Main.scala:95)
at $line3.$read$$iw$$iw.<init>(<console>:15)
at $line3.$read$$iw.<init>(<console>:42)
at $line3.$read.<init>(<console>:44)
at $line3.$read$.<init>(<console>:48)
at $line3.$read$.<clinit>(<console>)
at $line3.$eval$.$print$lzycompute(<console>:7)
at $line3.$eval$.$print(<console>:6)
at $line3.$eval.$print(<console>)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at scala.tools.nsc.interpreter.IMain$ReadEvalPrint.call(IMain.scala:786)
at scala.tools.nsc.interpreter.IMain$Request.loadAndRun(IMain.scala:1047)
at scala.tools.nsc.interpreter.IMain$WrappedRequest$$anonfun$loadAndRunReq$1.apply(IMain.scala:638)
at scala.tools.nsc.interpreter.IMain$WrappedRequest$$anonfun$loadAndRunReq$1.apply(IMain.scala:637)
at scala.reflect.internal.util.ScalaClassLoader$class.asContext(ScalaClassLoader.scala:31)
at scala.reflect.internal.util.AbstractFileClassLoader.asContext(AbstractFileClassLoader.scala:19)
at scala.tools.nsc.interpreter.IMain$WrappedRequest.loadAndRunReq(IMain.scala:637)
at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:569)
at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:565)
at scala.tools.nsc.interpreter.ILoop.interpretStartingWith(ILoop.scala:807)
at scala.tools.nsc.interpreter.ILoop.command(ILoop.scala:681)
at scala.tools.nsc.interpreter.ILoop.processLine(ILoop.scala:395)
at org.apache.spark.repl.SparkILoop$$anonfun$initializeSpark$1.apply$mcV$sp(SparkILoop.scala:38)
at org.apache.spark.repl.SparkILoop$$anonfun$initializeSpark$1.apply(SparkILoop.scala:37)
at org.apache.spark.repl.SparkILoop$$anonfun$initializeSpark$1.apply(SparkILoop.scala:37)
at scala.tools.nsc.interpreter.IMain.beQuietDuring(IMain.scala:214)
at org.apache.spark.repl.SparkILoop.initializeSpark(SparkILoop.scala:37)
at org.apache.spark.repl.SparkILoop.loadFiles(SparkILoop.scala:105)
at scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply$mcZ$sp(ILoop.scala:920)
at scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply(ILoop.scala:909)
at scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply(ILoop.scala:909)
at scala.reflect.internal.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:97)
at scala.tools.nsc.interpreter.ILoop.process(ILoop.scala:909)
at org.apache.spark.repl.Main$.doMain(Main.scala:68)
at org.apache.spark.repl.Main$.main(Main.scala:51)
at org.apache.spark.repl.Main.main(Main.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:738)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
17/01/28 22:03:28 INFO SparkUI: Stopped Spark web UI at http://fe80:0:0:0:2b9a:4521:a301:e9a5%10:4040
17/01/28 22:03:28 ERROR Utils: Uncaught exception in thread main
java.lang.NullPointerException
at org.apache.spark.scheduler.local.LocalSchedulerBackend.org$apache$spark$scheduler$local$LocalSchedulerBackend$$stop(LocalSchedulerBackend.scala:158)
at org.apache.spark.scheduler.local.LocalSchedulerBackend.stop(LocalSchedulerBackend.scala:137)
at org.apache.spark.scheduler.TaskSchedulerImpl.stop(TaskSchedulerImpl.scala:467)
at org.apache.spark.scheduler.DAGScheduler.stop(DAGScheduler.scala:1588)
at org.apache.spark.SparkContext$$anonfun$stop$8.apply$mcV$sp(SparkContext.scala:1826)
at org.apache.spark.util.Utils$.tryLogNonFatalError(Utils.scala:1283)
at org.apache.spark.SparkContext.stop(SparkContext.scala:1825)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:587)
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2313)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:868)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:860)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:860)
at org.apache.spark.repl.Main$.createSparkSession(Main.scala:95)
at $line3.$read$$iw$$iw.<init>(<console>:15)
at $line3.$read$$iw.<init>(<console>:42)
at $line3.$read.<init>(<console>:44)
at $line3.$read$.<init>(<console>:48)
at $line3.$read$.<clinit>(<console>)
at $line3.$eval$.$print$lzycompute(<console>:7)
at $line3.$eval$.$print(<console>:6)
at $line3.$eval.$print(<console>)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at scala.tools.nsc.interpreter.IMain$ReadEvalPrint.call(IMain.scala:786)
at scala.tools.nsc.interpreter.IMain$Request.loadAndRun(IMain.scala:1047)
at scala.tools.nsc.interpreter.IMain$WrappedRequest$$anonfun$loadAndRunReq$1.apply(IMain.scala:638)
at scala.tools.nsc.interpreter.IMain$WrappedRequest$$anonfun$loadAndRunReq$1.apply(IMain.scala:637)
at scala.reflect.internal.util.ScalaClassLoader$class.asContext(ScalaClassLoader.scala:31)
at scala.reflect.internal.util.AbstractFileClassLoader.asContext(AbstractFileClassLoader.scala:19)
at scala.tools.nsc.interpreter.IMain$WrappedRequest.loadAndRunReq(IMain.scala:637)
at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:569)
at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:565)
at scala.tools.nsc.interpreter.ILoop.interpretStartingWith(ILoop.scala:807)
at scala.tools.nsc.interpreter.ILoop.command(ILoop.scala:681)
at scala.tools.nsc.interpreter.ILoop.processLine(ILoop.scala:395)
at org.apache.spark.repl.SparkILoop$$anonfun$initializeSpark$1.apply$mcV$sp(SparkILoop.scala:38)
at org.apache.spark.repl.SparkILoop$$anonfun$initializeSpark$1.apply(SparkILoop.scala:37)
at org.apache.spark.repl.SparkILoop$$anonfun$initializeSpark$1.apply(SparkILoop.scala:37)
at scala.tools.nsc.interpreter.IMain.beQuietDuring(IMain.scala:214)
at org.apache.spark.repl.SparkILoop.initializeSpark(SparkILoop.scala:37)
at org.apache.spark.repl.SparkILoop.loadFiles(SparkILoop.scala:105)
at scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply$mcZ$sp(ILoop.scala:920)
at scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply(ILoop.scala:909)
at scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply(ILoop.scala:909)
at scala.reflect.internal.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:97)
at scala.tools.nsc.interpreter.ILoop.process(ILoop.scala:909)
at org.apache.spark.repl.Main$.doMain(Main.scala:68)
at org.apache.spark.repl.Main$.main(Main.scala:51)
at org.apache.spark.repl.Main.main(Main.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:738)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
17/01/28 22:03:28 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
17/01/28 22:03:28 INFO MemoryStore: MemoryStore cleared
17/01/28 22:03:28 INFO BlockManager: BlockManager stopped
17/01/28 22:03:28 INFO BlockManagerMaster: BlockManagerMaster stopped
17/01/28 22:03:28 WARN MetricsSystem: Stopping a MetricsSystem that is not running
17/01/28 22:03:28 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
17/01/28 22:03:28 INFO SparkContext: Successfully stopped SparkContext
java.lang.AssertionError: assertion failed: Expected hostname
at scala.Predef$.assert(Predef.scala:170)
at org.apache.spark.util.Utils$.checkHost(Utils.scala:931)
at org.apache.spark.util.RpcUtils$.makeDriverRef(RpcUtils.scala:31)
at org.apache.spark.executor.Executor.<init>(Executor.scala:121)
at org.apache.spark.scheduler.local.LocalEndpoint.<init>(LocalSchedulerBackend.scala:59)
at org.apache.spark.scheduler.local.LocalSchedulerBackend.start(LocalSchedulerBackend.scala:126)
at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:156)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:509)
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2313)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:868)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:860)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:860)
at org.apache.spark.repl.Main$.createSparkSession(Main.scala:95)
... 47 elided
<console>:14: error: not found: value spark
import spark.implicits._
^
<console>:14: error: not found: value spark
import spark.sql
^
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 2.1.0
/_/
Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_40)
Type in expressions to have them evaluated.
Type :help for more information.
推荐答案
好的，我似乎可以通过直接传递配置来解决它--conf spark.driver.host=localhost
OK, I seem to be able to get around it by passing configuration directly --conf spark.driver.host=localhost
所以我跑:
./bin/spark-shell --conf spark.driver.host=localhost
仍然有更好的解决方案，请让我知道.
Still if there is a better solution, please let me know.
Jacek Laskowski 已确认，这可能是目前唯一可用的解决方案.
Jacek Laskowski confirmed this is probably the only available solution for now.
这篇关于断开连接后，Spark无法在本地模式下启动[Spark中处理IPv6的可能错误?]的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
查看全文
