将数据附加到 HDFS Java 中的现有文件 [英] Append data to existing file in HDFS Java

问题描述

我无法将数据附加到 HDFS 中的现有文件.我希望如果文件存在则附加一行，如果不存在，则创建一个具有给定名称的新文件.

I'm having trouble to append data to an existing file in HDFS. I want that if the file exists then append a line, if not, create a new file with the name given.

这是我写入 HDFS 的方法.

Here's my method to write into HDFS.

if (!file.exists(path)){
   file.createNewFile(path);
}

FSDataOutputStream fileOutputStream = file.append(path); 
BufferedWriter br = new BufferedWriter(new OutputStreamWriter(fileOutputStream));
br.append("Content: " + content + "
");
br.close();

实际上这个方法会写入 HDFS 并创建一个文件，但正如我所提到的，不是追加.

Actually this method writes into HDFS and create a file but as I mention is not appending.

这是我测试我的方法的方式:

This is how I test my method:

RunTimeCalculationHdfsWrite.hdfsWriteFile("RunTimeParserLoaderMapperTest2", "Error message test 2.2", context, null);

第一个参数是文件名，第二个参数是消息，其他两个参数不重要.

The first param is the name of the file, the second the message and the other two params are not important.

所以有人知道我遗漏了什么或做错了什么吗?

So anyone have an idea what I'm missing or doing wrong?

推荐答案

其实可以追加到 HDFS 文件中:

Actually, you can append to a HDFS file:

从Client的角度来说，append操作首先调用DistributedFileSystem的append，这个操作会返回一个流对象FSDataOutputStream出来.如果Client需要向这个文件追加数据，可以调用out.write写，调用out.close关闭.

From the perspective of Client, append operation firstly calls append of DistributedFileSystem, this operation would return a stream object FSDataOutputStream out. If Client needs to append data to this file, it could calls out.write to write, and calls out.close to close.

我检查了 HDFS 源，有 DistributedFileSystem#append 方法:

I checked HDFS sources, there is DistributedFileSystem#append method:

 FSDataOutputStream append(Path f, final int bufferSize, final Progressable progress) throws IOException

有关详细信息，请参阅演示文稿.

For details, see presentation.

也可以通过命令行追加:

Also you can append through command line:

hdfs dfs -appendToFile <localsrc> ... <dst>

直接从标准输入添加行:

Add lines directly from stdin:

echo "Line-to-add" | hdfs dfs -appendToFile - <dst>

这篇关于将数据附加到 HDFS Java 中的现有文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！