使用 Hadoop 2.0 APIs 读写序列文件 [英] Reading and Writing Sequencefile using Hadoop 2.0 Apis

问题描述

我正在寻找一个使用新 API 读取和写入序列文件的示例.

I am looking for an example which is using the new API to read and write Sequence Files.

实际上我需要知道如何使用这些功能

Effectively I need to know how to use these functions

 createWriter(Configuration conf, org.apache.hadoop.io.SequenceFile.Writer.Option... opts) 

旧定义对我不起作用:

SequenceFile.createWriter( fs, conf, path, key.getClass(), value.getClass());

同样，我需要知道读取序列文件的代码是什么，因为以下内容已被弃用:

Similarly I need to know what will be the code for reading the Sequence file, as the follwoing is deprecated:

SequenceFile.Reader(fs, path, conf);

<小时>

这里是使用相同的方法-

---

Here is the way to use the same -

    String uri = args[0];
    Configuration conf = new Configuration();
    Path path = new Path( uri);

    IntWritable key = new IntWritable();
    Text value = new Text();

    CompressionCodec Codec = new GzipCodec();
    SequenceFile.Writer writer = null;
    Option optPath = SequenceFile.Writer.file(path);
    Option optKey = SequenceFile.Writer.keyClass(key.getClass());
    Option optVal = SequenceFile.Writer.valueClass(value.getClass());
    Option optCom = SequenceFile.Writer.compression(CompressionType.RECORD,  Codec);

        writer = SequenceFile.createWriter( conf, optPath, optKey, optVal, optCom);

推荐答案

public class SequenceFilesTest {
  @Test
  public void testSeqFileReadWrite() throws IOException {
    Configuration conf = new Configuration();
    FileSystem fs = FileSystem.getLocal(conf);
    Path seqFilePath = new Path("file.seq");
    SequenceFile.Writer writer = SequenceFile.createWriter(conf,
            Writer.file(seqFilePath), Writer.keyClass(Text.class),
            Writer.valueClass(IntWritable.class));

    writer.append(new Text("key1"), new IntWritable(1));
    writer.append(new Text("key2"), new IntWritable(2));

    writer.close();

    SequenceFile.Reader reader = new SequenceFile.Reader(conf,
            Reader.file(seqFilePath));

    Text key = new Text();
    IntWritable val = new IntWritable();

    while (reader.next(key, val)) {
        System.err.println(key + "	" + val);
    }

    reader.close();
  }
}

这篇关于使用 Hadoop 2.0 APIs 读写序列文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！