NoSuchMethodError使用Builder将Avro对象写入HDFS [英] NoSuchMethodError writing Avro object to HDFS using Builder

问题描述

将对象写入HDFS时出现此异常:

I'm getting this exception when writing an object to HDFS:

Exception in thread "main" java.lang.NoSuchMethodError: org.apache.avro.Schema$Parser.parse(Ljava/lang/String;[Ljava/lang/String;)Lorg/apache/avro/Schema;
        at com.blah.SomeType.<clinit>(SomeType.java:10)

它在生成的代码中引用的行是这样的:

The line it is referencing in the generated code is this:

public class SomeType extends org.apache.avro.specific.SpecificRecordBase implements org.apache.avro.specific.SpecificRecord {
  public static final org.apache.avro.Schema SCHEMA$ = new org.apache.avro.Schema.Parser().parse ..........

在我的代码中的调用是这样的:

And the call in my code is this:

val someTypeBuilder = SomeType.newBuilder()
      .setSomeField("some value")
      // set a whole load of other fields

在测试代码中调用newBuilder()根本不会导致任何问题.

Calling newBuilder() in test code causes no issues at all.

正在使用hadoop jar命令在HDFS节点上运行jar.

The jar is being run on an HDFS node using the hadoop jar command.

任何想法在这里可能出什么问题吗?

Any ideas what might be going wrong here?

推荐答案

1.7中没有它. 4 Schema.Parser文档.更新您的HDFS节点以使用更新版本的Avro，或自己加入Strings，就像

org.apache.avroSchema.Parser.parse(String s, String... more) was not introduced until Avro 1.7.5. Notice its absence in the 1.7.4 Schema.Parser documentation. Either update your HDFS node to use a more recent version of Avro, or join the Strings yourself like how the current 1.7 version of Schema does and use the method that takes a single String instead:

public static final org.apache.avro.Schema SCHEMA$ = new org.apache.avro.Schema.Parser().parse(String.join("", prevArg1, prevArg2));

我从未尝试过在Hadoop安装上升级Avro，但是我想您可能只是找到jar包，如果是次要版本升级，则可以手动替换它们.

I've never tried upgrading Avro on a Hadoop installation, but I would guess you could probably just find the jars and replace them manually if it's a minor version upgrade.

~/dev/hadoop/hadoop-2.7.1$ find . -name '*avro*.jar'
./share/hadoop/tools/lib/avro-1.7.4.jar
./share/hadoop/httpfs/tomcat/webapps/webhdfs/WEB-INF/lib/avro-1.7.4.jar
./share/hadoop/common/lib/avro-1.7.4.jar
./share/hadoop/mapreduce/lib/avro-1.7.4.jar
./share/hadoop/kms/tomcat/webapps/kms/WEB-INF/lib/avro-1.7.4.jar

尝试下载 Avro 1.7 .7 ，然后将所有找到的jar替换为下载的jar.如果出现问题，我还将备份Hadoop安装.

Try downloading Avro 1.7.7 and replacing all those found jars with downloaded one. I would also back up the Hadoop installation in case it goes awry.

这篇关于NoSuchMethodError使用Builder将Avro对象写入HDFS的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！