将图像/视频存储到 Hadoop HDFS [英] Store images/videos into Hadoop HDFS
问题描述
我想将一些视频/图像存储到 Hadoop HDFS,但我听说HDFS 只接受像文本这样的文件.
I would like to store some videos/images into Hadoop HDFS, but I heard that HDFS accepts only files like as a text.
可以肯定的是,我们可以将视频/图像存储到 HDFS 中吗?如果是,那么这样做的方法或步骤是什么?
To be sure, can we store videos/images into HDFS? If yes, what's the way or the steps to follow to do that?
推荐答案
完全可以不做任何额外的事情.Hadoop 为我们提供了读/写二进制文件的工具.因此,几乎任何可以转换为字节的东西都可以存储到 HDFS(图像、视频等)中.为此,Hadoop 提供了称为SequenceFiles 的内容.SequenceFile 是一个由二进制键/值对组成的平面文件.SequenceFile 提供了 Writer、Reader 和 Sorter 类,分别用于写入、读取和排序.因此,您可以将图像/视频文件转换为 SeuenceFile 并将其存储到 HDFS 中.这是一小段代码,它将获取图像文件并将其转换为 SequenceFile,其中文件名是键,图像内容是值:
It is absolutely possible without doing anything extra. Hadoop provides us the facility to read/write binary files. So, practically anything which can be converted into bytes can be stored into HDFS(images, videos etc). To do that Hadoop provides something called as SequenceFiles. SequenceFile is a flat file consisting of binary key/value pairs. The SequenceFile provides a Writer, Reader and Sorter classes for writing, reading and sorting respectively. So, you could convert your image/video file into a SeuenceFile and store it into the HDFS. Here is small piece of code that will take an image file and convert it into a SequenceFile, where name of the file is the key and image content is the value :
public class ImageToSeq {
public static void main(String args[]) throws Exception {
Configuration confHadoop = new Configuration();
confHadoop.addResource(new Path("/hadoop/projects/hadoop-1.0.4/conf/core-site.xml"));
confHadoop.addResource(new Path("/hadoop/projects/hadoop-1.0.4/conf/hdfs-site.xml"));
FileSystem fs = FileSystem.get(confHadoop);
Path inPath = new Path("/mapin/1.png");
Path outPath = new Path("/mapin/11.png");
FSDataInputStream in = null;
Text key = new Text();
BytesWritable value = new BytesWritable();
SequenceFile.Writer writer = null;
try{
in = fs.open(inPath);
byte buffer[] = new byte[in.available()];
in.read(buffer);
writer = SequenceFile.createWriter(fs, confHadoop, outPath, key.getClass(),value.getClass());
writer.append(new Text(inPath.getName()), new BytesWritable(buffer));
}catch (Exception e) {
System.out.println("Exception MESSAGES = "+e.getMessage());
}
finally {
IOUtils.closeStream(writer);
System.out.println("last line of the code....!!!!!!!!!!");
}
}
}
如果您的目的只是按原样转储文件,您可以简单地这样做:
And if your intention is to just dump the files as it is, you could simply do this :
bin/hadoop fs -put /src_image_file /dst_image_file
如果您的目的不仅仅是存储文件,您可能会发现 HIPI 很有用.HIPI 是 Hadoop 的 MapReduce 框架的库,它提供了一个 API,用于在分布式计算环境中执行图像处理任务.
And if your intent is more than just storing the files, you might find HIPI useful. HIPI is a library for Hadoop's MapReduce framework that provides an API for performing image processing tasks in a distributed computing environment.
HTH
这篇关于将图像/视频存储到 Hadoop HDFS的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!