星火流的StreamingContext错误 [英] Spark Streaming StreamingContext error

问题描述

我是一个Java老将谁是努力学习斯卡拉+星火流。
我下载的基于Eclipse的IDE斯卡拉星火+核心+罐子星火流罐子都2.10和尝试的例子 - 我得到的错误：

  VAL SSC =新的StreamingContext（CONF，秒（1））;
 

  

描述资源路径位置类型糟糕的符号引用。一个
  在StreamingContext.class签名是指长期的conf封装
  org.apache.hadoop这是不可用的。它可以是完全缺失
  从目前的类路径，或者在类路径的版本可能
  编译时的版本不兼容使用
  StreamingContext.class。 Lab.scala / AirStream icon免费1.0.1 / src目录线10斯卡拉问题

有什么，我错过了这里？所有SparkContext没有错误，但在的StreamingContext收到此错误人的时间。

解决方案

请确保在类路径中的Hadoop版本匹配的火花流罐始建兑之一。也可能有一些相关性的火花流预期在集群环境中提供;如果是这样你将需要在Eclipse运行时手动添加到classpath中。

I'm a Java veteran who's trying to learn Scala + Spark Streaming. I downloaded Eclipse-based Scala IDE + Spark core jar + Spark Streaming jar both 2.10 and try out the example - I'm getting the error:

val ssc = new StreamingContext(conf, Seconds(1));

Description Resource Path Location Type bad symbolic reference. A signature in StreamingContext.class refers to term conf in package org.apache.hadoop which is not available. It may be completely missing from the current classpath, or the version on the classpath might be incompatible with the version used when compiling StreamingContext.class. Lab.scala /AirStream/src line 10 Scala Problem

Is there something that I missed here? all SparkContext has no error but StreamingContext is getting this error al the time.

解决方案

Make sure the version of hadoop on the classpath matches the one that the spark streaming jar was built against. There might also be some dependencies that spark streaming expects to be provided by the cluster environment; if so you will need to add them manually to the classpath when running in eclipse.

这篇关于星火流的StreamingContext错误的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！