在Hadoop上运行jar时，本机快照库是什么 [英] What's the native snappy library when running jar with Hadoop

问题描述

当我在Centos 6.4中运行MapReduce jar时，出现以下错误提示.

There is an Error as notice below when I ran a MapReduce jar in Centos 6.4 .

Hadoop版本为2.6.0(用于64位).

Hadoop Version is 2.6.0 for 64 bit.

MapReduce失败，我该如何解决?

The MapReduce failed,how can I solve this?

Error: java.lang.RuntimeException: native snappy library not available: this version of libhadoop was built without snappy support.
    at org.apache.hadoop.io.compress.SnappyCodec.checkNativeCodeLoaded(SnappyCodec.java:64)
    at org.apache.hadoop.io.compress.SnappyCodec.getCompressorType(SnappyCodec.java:133)
    at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:148)
    at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:163)
    at org.apache.hadoop.io.SequenceFile$Writer.init(SequenceFile.java:1199)
    at org.apache.hadoop.io.SequenceFile$Writer.<init>(SequenceFile.java:1092)
    at org.apache.hadoop.io.SequenceFile$BlockCompressWriter.<init>(SequenceFile.java:1442)
    at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:275)
    at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:528)
    at org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat.getSequenceWriter(SequenceFileOutputFormat.java:64)
    at org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat.getRecordWriter(SequenceFileOutputFormat.java:75)
    at org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.<init>(ReduceTask.java:540)
    at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:614)
    at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)

推荐答案

您需要为本地库编译hadoop源代码. 此处.

You need to compile hadoop source code for native libraries. Step by step instructions are provided here.

视频介绍了验证，使用maven和本机库编译代码.

The video covers validation, compiling the code using maven and using the native libraries.

这篇关于在Hadoop上运行jar时，本机快照库是什么的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！