Maven对Spark的StreamingQuery的依赖 [英] Maven dependency for Spark's StreamingQuery

问题描述

我有一个编译时错误:

无法解析符号"spark"

对于以下代码:

import org.apache.spark.sql.streaming.StreamingQuery;

我使用InteliJ.

pom.xml:

...
<dependencies>
        <dependency>
            <groupId>com.sparkjava</groupId>
            <artifactId>spark-core</artifactId>
            <version>2.5</version>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-sql_2.11</artifactId>
            <version>2.4.0</version>
        </dependency>
    </dependencies>

我注意到Maven抱怨spark-sql依赖项:

解决方案

我可以确认spark-sql_2.11:2.4.0中存在接口org.apache.spark.sql.streaming.StreamingQuery，并且源代码可用此处: https://github.com/apache/spark/blob/v2.4.0/sql/core/src/main/scala/org/apache/spark/sql/streaming/StreamingQuery.scala

我认为编译错误与源代码中的导入有关，与某种类型无关(您尚未附加发生故障的文件的源代码，因此我只能在此处进行推测)

因此，就分辨率而言:

1. 尝试在没有智能的情况下从命令行运行mvn clean compile .如果成功，请检查intelliJ集成(也许重新导入将解决此问题).

2. 如果没有，请检查pom.xml，也许是您错误地更新了<dependencyManagement>部分.

3. 从本地Maven存储库中删除jar并重试，有时有可能jar被下载损坏

更新1 从现在开始，我们知道命令行中的mvn clean compile已成功运行， 这一定是在intelliJ中打开项目的问题.

因此，我建议您执行以下操作:

1. 关闭当前项目:文件->关闭项目
2. 文件->打开...现在找到项目文件夹，然后(选择这是很重要的)选择pom.xml. IntelliJ将询问您是否要删除以前找到的项目文件-回答是"，它将重新构建内部依赖项数据结构. 然后尝试运行程序并更新问题是否存在.

I have a compile-time error:

Cannot resolve symbol 'spark'

For the following code:

import org.apache.spark.sql.streaming.StreamingQuery;

I use InteliJ.

pom.xml:

...
<dependencies>
        <dependency>
            <groupId>com.sparkjava</groupId>
            <artifactId>spark-core</artifactId>
            <version>2.5</version>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-sql_2.11</artifactId>
            <version>2.4.0</version>
        </dependency>
    </dependencies>

Edit:

I've noticed that Maven complains about the spark-sql dependency:

解决方案

I can confirm that the interface org.apache.spark.sql.streaming.StreamingQuery exists in spark-sql_2.11:2.4.0 with the source code available here: https://github.com/apache/spark/blob/v2.4.0/sql/core/src/main/scala/org/apache/spark/sql/streaming/StreamingQuery.scala

I assume that the compilation error is related to the import in the source code, and not to some type (you haven't attached the source code of the file that has a failure, so I can only speculate here)

So, in terms of resolution:

1. try running mvn clean compile from command line without intelliJ. If it succeeds - check intelliJ integration (maybe re-import will fix the issue).

2. If it doesn't - check the pom.xml maybe you've updated a <dependencyManagement> section by mistake.

3. Remove the jars of spark from local maven repository and retry, sometimes its possible that the jar will get downloaded corrupted

Update 1 Since now we know that mvn clean compile from the command line runs successfully, this must be an issue of opening project in intelliJ.

So I suggest to do the following:

1. Close the current project: File --> Close Project
2. File --> Open... Now find the project folder and (this is important) pick pom.xml. IntelliJ will ask whether you want to delete the previously found project files - respond "yes" and it will re-build an internal dependencies data structures. Then Try to run the program and update whether the issue exists.

这篇关于Maven对Spark的StreamingQuery的依赖的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！