将图像/视频存储到Hadoop HDFS中 [英] Store images/videos into Hadoop HDFS

问题描述

我想将一些视频/图片存储到Hadoop HDFS ，但我听说过HDFS只接受像文本一样的文件。

可以肯定的是，我们可以将视频/图像存储到HDFS吗？如果是的话，那么做什么的方式或步骤呢？ 解决方案

绝对有可能不做任何额外的事情。 Hadoop为我们提供了读取/写入二进制文件的工具。所以，实际上任何可以转换为字节的东西都可以存储到HDFS（图像，视频等）中。为此，Hadoop提供了一种称为 SequenceFiles 的东西。 SequenceFile是一个由二进制键/值对组成的平面文件。 SequenceFile提供了一个Writer，Reader和Sorter类，分别用于写入，读取和排序。因此，您可以将图像/视频文件转换为SeuenceFile并将其存储到HDFS中。这是一小段代码，它将拍摄一个图像文件并将其转换为SequenceFile，其中文件的名称是关键字，图像内容是该值：

<$ p $ （$ {
}} public static void main（String args []）抛出异常{

配置confHadoop = new Configuration（）;
confHadoop.addResource（new Path（/ hadoop / projects / hadoop-1.0.4 / conf / core-site.xml））;
confHadoop.addResource（new Path（/ hadoop / projects / hadoop-1.0.4 / conf / hdfs-site.xml））;
FileSystem fs = FileSystem.get（confHadoop）;
路径inPath =新路径（/ mapin / 1.png）;
路径outPath =新路径（/ mapin / 11.png）;
FSDataInputStream in = null;
Text key = new Text（）;
BytesWritable value = new BytesWritable（）;
SequenceFile.Writer writer = null;
尝试{
in = fs.open（inPath）;
字节缓冲区[] =新字节[in.available（）];
in.read（buffer）;
writer = SequenceFile.createWriter（fs，confHadoop，outPath，key.getClass（），value.getClass（））;
writer.append（new Text（inPath.getName（）），new BytesWritable（buffer））;
} catch（Exception e）{
System.out.println（Exception MESSAGES =+ e.getMessage（））;
}
finally {
IOUtils.closeStream（writer）;
System.out.println（代码的最后一行.... !!!!!!!!!!）;



$ / code>

如果你的目的是只需转储文件就可以了：

  bin / hadoop fs -put / src_image_file / dst_image_file 
  

如果您的意图不仅仅是存储文件，您可能会发现 HIPI 有用。 HIPI是Hadoop的MapReduce框架的一个库，它提供了一个用于在分布式计算环境中执行图像处理任务的API。

HTH

I would like to store some videos/images into Hadoop HDFS, but I heard that HDFS accepts only files like as a text.

To be sure, can we store videos/images into HDFS? If yes, what's the way or the steps to follow to do that?

解决方案

It is absolutely possible without doing anything extra. Hadoop provides us the facility to read/write binary files. So, practically anything which can be converted into bytes can be stored into HDFS(images, videos etc). To do that Hadoop provides something called as SequenceFiles. SequenceFile is a flat file consisting of binary key/value pairs. The SequenceFile provides a Writer, Reader and Sorter classes for writing, reading and sorting respectively. So, you could convert your image/video file into a SeuenceFile and store it into the HDFS. Here is small piece of code that will take an image file and convert it into a SequenceFile, where name of the file is the key and image content is the value :

public class ImageToSeq {
    public static void main(String args[]) throws Exception {

        Configuration confHadoop = new Configuration();     
        confHadoop.addResource(new Path("/hadoop/projects/hadoop-1.0.4/conf/core-site.xml"));
        confHadoop.addResource(new Path("/hadoop/projects/hadoop-1.0.4/conf/hdfs-site.xml"));   
        FileSystem fs = FileSystem.get(confHadoop);
        Path inPath = new Path("/mapin/1.png");
        Path outPath = new Path("/mapin/11.png");
        FSDataInputStream in = null;
        Text key = new Text();
        BytesWritable value = new BytesWritable();
        SequenceFile.Writer writer = null;
        try{
            in = fs.open(inPath);
            byte buffer[] = new byte[in.available()];
            in.read(buffer);
            writer = SequenceFile.createWriter(fs, confHadoop, outPath, key.getClass(),value.getClass());
            writer.append(new Text(inPath.getName()), new BytesWritable(buffer));
        }catch (Exception e) {
            System.out.println("Exception MESSAGES = "+e.getMessage());
        }
        finally {
            IOUtils.closeStream(writer);
            System.out.println("last line of the code....!!!!!!!!!!");
        }
    }
}

And if your intention is to just dump the files as it is, you could simply do this :

bin/hadoop fs -put /src_image_file /dst_image_file

And if your intent is more than just storing the files, you might find HIPI useful. HIPI is a library for Hadoop's MapReduce framework that provides an API for performing image processing tasks in a distributed computing environment.

HTH

这篇关于将图像/视频存储到Hadoop HDFS中的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！