通过 SFTP 从服务器读取大型动态文件的结尾 [英] Reading end of huge and dynamic file via SFTP from server

问题描述

我正在尝试找到一种方法，从服务器通过 SFTP 读取巨大的动态日志文件的末尾(例如从末尾开始的 20-30 行)，并将该点保存到我读取的位置，如果我需要更多行，从这一点开始阅读更多内容.

I am trying to find a way to read just end of huge and dynamic log file (like 20-30 lines from end) via SFTP from server and to save the point until where I read, and if I need more lines, to read more from this point upper.

我尝试过的一切都需要很长时间，我尝试在机器上复制这个文件，然后使用 ReversedLinesFileReader 从末尾读取，因为这个方法需要 File 对象，当通过 SFTP 时，你只会得到 InputStream，下载文件需要很多时间.

Everything I've tried takes too long time, I've tried to copy this file on machine and after this to read from end using ReversedLinesFileReader because this method need the File object, when via SFTP you will get only InputStream, takes a lot to download file.

还尝试计算行数并从 n 行读取，但也花费了太长时间并抛出异常，因为此时文件已被修改.另一种方法是我尝试通过 SSH 连接并使用 tail -100 并获得所需的结果，但只是一次，因为下次我也会获得新日志，但我需要走上坡路.有没有一种快速的方法来获取文件的结尾并保存该点并稍后阅读更多的内容?有什么想法吗?

Also tried to count lines and to read from n line but also takes too long and throws exception because sometime in this time file is modified. Another way I tried to connect via SSH and used tail -100 and get the desired result, but just for one time, because next time I will get also new logs, but I need to go upper. Is there a fast way to get the end of file and to save the point and to read more upper of this point later? Any idea?

推荐答案

你不说你用的是什么 SFTP 库，但使用最广泛的 Java SSH/SFTP 库是 JSch，所以我假设您正在使用它.

You don't say what SFTP library you're using, but the most widely used Java SSH/SFTP library is JSch, so I'll assume you're using that.

SFTP 协议具有执行随机的操作-访问远程文件上的 I/O.不幸的是，JSch SFTP 客户端并未公开所有操作.但是，它确实有 get 操作(用于从远程服务器获取文件)允许跳过远程文件的第一部分.例如，您可以使用这些操作之一读取文件的最后 10 KB.

The SFTP protocol has operations to perform random-access I/O on remote files. Unfortunately, the JSch SFTP client doesn't expose the full range of operations. However, it does have versions of the get operation (for getting a file from the remote server) which permit skipping over the first part of the remote file. You can use one of these operations to read for example the last 10 KB of a file.

几个 JSch get 操作返回一个 InputStream.您可以从输入流中读取远程文件的内容.如果要逐行访问远程文件，可以将其转换为 Reader 使用 InputStreamReader.

Several of the JSch get operations return an InputStream. You can read the contents of the remote file from the input stream. If you want to access the remote file line-by-line, you can convert it to Reader using InputStreamReader.

因此，进程可能会执行以下操作:

So, a process might do the following:

1. 调用 stat() 获取远程文件的大小.
2. 找出您要从文件中的哪个位置开始读取.您可以跟踪上次停止阅读的位置，也可以根据您愿意下载的数据量以及最后 20-30 行的预期大小(以字节为单位)进行猜测.
3. 调用 get() 开始阅读.
4. 处理从 get() 调用返回的 InputStream 读取的数据.

1. Call stat() on the remote file to get its size.
2. Figure out where in the file you want to start reading from. You could keep track of where you stopped reading last time, or you could guess based on the amount of data you're willing to download and the expected size in bytes of these last 20-30 lines.
3. Call get() to start reading it.
4. Process data read from the InputStream returned by the get() call.

这篇关于通过 SFTP 从服务器读取大型动态文件的结尾的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！