Spark's StreamingQuery 的 Maven 依赖 [英] Maven dependency for Spark's StreamingQuery

问题描述

我有一个编译时错误:

无法解析符号火花"

对于以下代码:

import org.apache.spark.sql.streaming.StreamingQuery;

我使用 InteliJ.

I use InteliJ.

pom.xml:

...
<dependencies>
        <dependency>
            <groupId>com.sparkjava</groupId>
            <artifactId>spark-core</artifactId>
            <version>2.5</version>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-sql_2.11</artifactId>
            <version>2.4.0</version>
        </dependency>
    </dependencies>

我注意到 Maven 抱怨 spark-sql 依赖:

I've noticed that Maven complains about the spark-sql dependency:

推荐答案

我可以确认spark-sql_2.11:2.4.0中存在接口org.apache.spark.sql.streaming.StreamingQuery，源码可用这里:https://github.com/apache/spark/blob/v2.4.0/sql/core/src/main/scala/org/apache/spark/sql/streaming/StreamingQuery.scala

I can confirm that the interface org.apache.spark.sql.streaming.StreamingQuery exists in spark-sql_2.11:2.4.0 with the source code available here: https://github.com/apache/spark/blob/v2.4.0/sql/core/src/main/scala/org/apache/spark/sql/streaming/StreamingQuery.scala

我假设编译错误与源代码中的导入有关，而不是与某种类型有关(您没有附上失败的文件的源代码，所以我只能在这里推测)

I assume that the compilation error is related to the import in the source code, and not to some type (you haven't attached the source code of the file that has a failure, so I can only speculate here)

因此，就分辨率而言:

1. 尝试在没有 intelliJ 的情况下运行 mvn clean compile 从命令行.如果成功 - 检查 intelliJ 集成(也许重新导入可以解决问题).

1. try running mvn clean compile from command line without intelliJ. If it succeeds - check intelliJ integration (maybe re-import will fix the issue).

如果没有 - 检查 pom.xml 可能是您错误地更新了 部分.

If it doesn't - check the pom.xml maybe you've updated a <dependencyManagement> section by mistake.

从本地 Maven 存储库中删除 Spark 的 jar 包并重试，有时可能下载的 jar 包已损坏

Remove the jars of spark from local maven repository and retry, sometimes its possible that the jar will get downloaded corrupted

更新 1既然现在我们知道命令行中的 mvn clean compile 运行成功，这一定是intelliJ中打开项目的问题.

Update 1 Since now we know that mvn clean compile from the command line runs successfully, this must be an issue of opening project in intelliJ.

所以我建议做以下事情:

So I suggest to do the following:

1. 关闭当前项目:文件 --> 关闭项目
2. File --> Open... 现在找到项目文件夹并(这很重要)选择 pom.xml.IntelliJ 会询问您是否要删除之前找到的项目文件 - 回答是"，它将重新构建内部依赖项数据结构.然后尝试运行程序并更新问题是否存在.

1. Close the current project: File --> Close Project
2. File --> Open... Now find the project folder and (this is important) pick pom.xml. IntelliJ will ask whether you want to delete the previously found project files - respond "yes" and it will re-build an internal dependencies data structures. Then Try to run the program and update whether the issue exists.

这篇关于Spark's StreamingQuery 的 Maven 依赖的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！