NoSuchMethodError：org.apache.hadoop.conf.Configuration.reloadExistingConfigurations()V [英] NoSuchMethodError: org.apache.hadoop.conf.Configuration.reloadExistingConfigurations()V

问题描述

我在我的项目中使用的是带Detla版本0.8.0的<；spark.version>；3.0.2<；/spark.version>；。 并与

一起运行

export SPARK_HOME=/pkg/spark-3.0.2-bin-hadoop2.7-hive1.2
$SPARK_HOME/bin/spark-submit 
--master yarn 
--deploy-mode cluster 

--packages io.delta:delta-core_2.12:0.8.0,org.apache.hadoop:hadoop-common:2.9.2,org.apache.hadoop:hadoop-aws:2.9.2,org.apache.hudi:hudi-spark-bundle_2.12:0.6.0 

我遇到错误

 User class threw exception: java.lang.NoSuchMethodError: org.apache.hadoop.conf.Configuration.reloadExistingConfigurations()V
at org.apache.hadoop.fs.s3a.S3AFileSystem.addDeprecatedKeys(S3AFileSystem.java:183)
at org.apache.hadoop.fs.s3a.S3AFileSystem.<clinit>(S3AFileSystem.java:187)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:2134)
at org.apache.spark.sql.delta.sources.DeltaDataSource.createRelation(DeltaDataSource.scala:171)
at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:354)
at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:297)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:232)

这里出了什么问题？？您知道如何修复它吗？

已尝试

export SPARK_HOME=/pkg/spark-3.0.2-bin-hadoop2.9.1-custom

出现错误

Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: ShuffleMapStage 4 ($anonfun$call$1 at DatabricksLogging.scala:77) has failed the maximum allowable number of times: 4. Most recent failure reason: org.apache.spark.shuffle.FetchFailedException: java.lang.IllegalArgumentException: Unknown message type: 9     at org.apache.spark.network.shuffle.protocol.BlockTransferMessage$Decoder.fromByteBuffer(BlockTransferMessage.java:71)  at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)    ... 1 more 
    at org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:2059)
    at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:2007)
    at org.apache.spark.scheduler.DAGScheduler.handleTaskCompletion(DAGScheduler.scala:1602)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2236)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2188)
    at org.apache.spark.sql.delta.DeltaLog$.recordDeltaOperation(DeltaLog.scala:368)
    at org.apache.spark.sql.delta.DeltaLog$$anon$3.call(DeltaLog.scala:470)
    at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2379)

推荐答案

您的Spark是使用Hadoop2.7编译的，但您正在尝试使用Hadoop2.9执行代码。从--packages命令中删除Hadoop 2.9坐标

这篇关于NoSuchMethodError：org.apache.hadoop.conf.Configuration.reloadExistingConfigurations()V的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！