是否可以从多个客户端并行附加到 HDFS 文件? [英] Is it possible to append to HDFS file from multiple clients in parallel?

问题描述

基本上整个问题都在标题中.我想知道是否可以同时从多台计算机附加到位于 HDFS 上的文件?类似于存储由多个进程不断产生的事件流.顺序不重要.

Basically whole question is in the title. I'm wondering if it's possible to append to file located on HDFS from multiple computers simultaneously? Something like storing stream of events constantly produced by multiple processes. Order is not important.

我记得在 Google 的一次技术演示中听说 GFS 支持此类附加功能，但尝试使用 HDFS(使用常规文件 append() 或使用 SequenceFile)进行的一些有限测试似乎不起作用.

I recall hearing on one of the Google tech presentations that GFS supports such append functionality but trying some limited testing with HDFS (either with regular file append() or with SequenceFile) doesn't seems to work.

谢谢，

推荐答案

我认为 HDFS 不可能做到这一点.尽管您不关心记录的顺序，但您确实关心文件中字节的顺序.您不希望写入器 A 写入部分记录，然后被写入器 B 损坏.这是 HDFS 自行解决的一个难题，因此它不会.

I don't think that this is possible with HDFS. Even though you don't care about the order of the records, you do care about the order of the bytes in the file. You don't want writer A to write a partial record that then gets corrupted by writer B. This is a hard problem for HDFS to solve on its own, so it doesn't.

为每个作者创建一个文件.将所有文件传递给任何需要读取此数据的 MapReduce 工作器.这要简单得多，并且适合 HDFS 和 Hadoop 的设计.如果非 MapReduce 代码需要将这些数据作为一个流读取，那么要么按顺序流式传输每个文件，要么编写一个非常快速的 MapReduce 作业来合并文件.

Create a file per writer. Pass all the files to any MapReduce worker that needs to read this data. This is much simpler and fits the design of HDFS and Hadoop. If non-MapReduce code needs to read this data as one stream then either stream each file sequentially or write a very quick MapReduce job to consolidate the files.

这篇关于是否可以从多个客户端并行附加到 HDFS 文件?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！