写入Avro数据文件 [英] Writing to Avro Data file

问题描述

以下代码仅将数据写入avro格式，并从同样写入的avro文件中读取并显示相同的数据.我只是在尝试Hadoop最终指导书中的示例.我第一次能够执行.然后我得到了以下错误.它确实是第一次工作.所以我不确定我正在犯什么错误.

The following code simply writes data into avro format and reads and displays the same from the avro file written too. I was just trying out the example in the Hadoop definitive guide book. I was able to execute this first time. Then I got the following error. It did work for the first time. So I am not sure wat mistake i am making.

这是一个例外:

Exception in thread "main" java.io.EOFException: No content to map to Object due to end of input
    at org.codehaus.jackson.map.ObjectMapper._initForReading(ObjectMapper.java:2173)
    at org.codehaus.jackson.map.ObjectMapper._readValue(ObjectMapper.java:2106)
    at org.codehaus.jackson.map.ObjectMapper.readTree(ObjectMapper.java:1065)
    at org.codehaus.jackson.map.ObjectMapper.readTree(ObjectMapper.java:1040)
    at org.apache.avro.Schema.parse(Schema.java:895)
    at org.avro.example.SimpleAvro.AvroExample.avrocreate(AvroDataExample.java:23)
    at org.avro.example.SimpleAvro.AvroDataExample.main(AvroDataExample.java:55)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:156)

这是代码:

package org.avro.example.SimpleAvro;

import java.io.File;
import java.io.IOException;

import org.apache.avro.Schema;
import org.apache.avro.file.DataFileReader;
import org.apache.avro.file.DataFileWriter;
import org.apache.avro.generic.GenericData;
import org.apache.avro. generic.GenericDatumReader;
import org.apache.avro.generic.GenericDatumWriter;
import org.apache.avro.generic.GenericRecord;
import org.apache.avro.io.DatumReader;
import org.apache.avro.io.DatumWriter;

class AvroExample{

    AvroExample(){

    }
    void avrocreate() throws Exception{

        Schema schema=Schema.parse(getClass().getResourceAsStream("Pair.avsc"));

        GenericRecord datum=new GenericData.Record(schema);
        datum.put("left", "L");
        datum.put("right", "R");

        File file=new File("data.avro");
        DatumWriter<GenericRecord> writer=new GenericDatumWriter<GenericRecord>(schema);
        DataFileWriter<GenericRecord> dataFileWriter=new DataFileWriter<GenericRecord>(writer);
        dataFileWriter.create(schema, file);
        dataFileWriter.append(datum);
        dataFileWriter.close();

        System.out.println("Written to avro data file");
        //reading from the avro data file

        DatumReader<GenericRecord> reader= new GenericDatumReader<GenericRecord>();
        DataFileReader<GenericRecord> dataFileReader=new DataFileReader<GenericRecord>(file,reader);
        GenericRecord result=dataFileReader.next();
        System.out.println("data" + result.get("left").toString());

        result=dataFileReader.next();
        System.out.println("data :" + result.get("left").toString());


    }

}
public class AvroDataExample {
    public static void main(String args[])throws Exception{

        AvroExample a=new AvroExample();
        a.avrocreate();
    }



}

以下是Pair.avsc文件[在本书的示例代码中给出]

The following is the Pair.avsc file [ given in the book's example code]

{
  "type": "record",
  "name": "Pair",
  "doc": "A pair of strings.",
  "fields": [
    {"name": "left", "type": "string"},
    {"name": "right", "type": "string"}
  ]
}

推荐答案

您可能没有正确读取架构文件.我怀疑这是问题所在，因为堆栈跟踪显示它无法解析架构:

You are probably not reading the schema file correctly. I suspect this is the problem because the stack trace shows that it is failing to parse the schema:

Exception in thread "main" java.io.EOFException: No content to map to Object due to end of input
    at org.codehaus.jackson.map.ObjectMapper._initForReading(ObjectMapper.java:2173)
    at org.codehaus.jackson.map.ObjectMapper._readValue(ObjectMapper.java:2106)
    at org.codehaus.jackson.map.ObjectMapper.readTree(ObjectMapper.java:1065)
    at org.codehaus.jackson.map.ObjectMapper.readTree(ObjectMapper.java:1040)
    at org.apache.avro.Schema.parse(Schema.java:895)

除非您的环境设置正确，否则从资源"中读取文件充满了问题.另外，由于您曾经提到它曾经工作过一次，因此您可能只是为第二次运行更改了一些环境设置(例如工作目录).

Reading files from "resources" is fraught with problems unless you have your environment set up just right. Also, since you mentioned that it worked once before, you may just have changed some environmental setting (such as working directory) for the second runs.

尝试将架构字符串复制粘贴到String变量中，然后

Try copy-pasting the schema string into a String variable and parse it directly rather than using the resource loader:

String schemaJson = "paste schema here (and fix quotes)";
Schema schema = Schema.parse(schemaJson);
GenericRecord datum = new GenericData.Record(schema);
...

这篇关于写入Avro数据文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！