用hadoop进行图像处理 [英] image processing with hadoop

问题描述

如何在hadoop中读取视频帧？为解决Hadoop中的专业文件格式（如视频）必须编写一个自定义的InputFormat和RecordReader来理解如何将视频文件转换为分割（InputFormat），然后将分割读入值（RecordReader）。这是一项不平凡的任务，需要一些关于Hadoop如何处理数据分割的中间知识。我强烈推荐O'Reilly编写的Tom White的Hadoop权威指南，以及 http://www.cloudera.com上的视频。 （完全披露：我为Cloudera工作）。请注意，视频格式通常会被压缩，这会变得更加复杂，因为InputSplits（由InputFormat创建）是简单的字节偏移到文件中（通常）。从 http：//hadoop.apache开始.org / common / docs / current / api / org / apache / hadoop / mapred / InputFormat.html

总结：InputFormat知道如何生成（通常）在64MB到128MB之间的InputSplit对象列表，并且 NOT 遵守帧的概念。然后使用RecordReader从InputSplit中读取框架以创建地图缩减作业可以处理的值对象。如果您想生成视频输出，您还需要编写一个自定义OutputFormat。

希望这有助于您。

How to read video frames in hadoop?

解决方案

To process specialized file formats (such as video) in Hadoop, you'd have to write a custom InputFormat and RecordReader that understands how to turn a video file into splits (the InputFormat) and then read splits into values (the RecordReader). This is a non-trivial task and requires some intermediate knowledge of how Hadoop handles the splitting of data. I highly recommend Tom White's Hadoop the Definitive Guide book by O'Reilly as well as the videos on http://www.cloudera.com. (Full disclosure: I work for Cloudera.)

Keep in mind that video formats are generally compressed which gets even more complicated because InputSplits (created by an InputFormat) are simple byte offsets into the file (normally). Start with http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapred/InputFormat.html

To summarize: InputFormat knows how to generate a list of InputSplit objects that are (usually) between 64MB and 128MB and do NOT respect the notion of frames. The RecordReader then is used to read frames out of a InputSplit to create value objects that the map reduce job can process. If you want to generate video output you'll also need to write a custom OutputFormat.

Hope this helps.

这篇关于用hadoop进行图像处理的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！