使用Node.js和SSH2从SFTP服务器读取文件 [英] Reading file from SFTP server using Node.js and SSH2

问题描述

我在Node.js中使用读取流时遇到了一个非常奇怪的问题。我正在使用SSH2在我和sftp服务器之间创建一个sftp连接。然后我尝试从sftp流创建一个读取流。从读取流发出的数据事件中，我将数据附加到数组。当读取流的'close'事件发生时，我调用Buffer.concat来创建concat，将我检索到的所有数据块连接到一个缓冲区中。这是在堆栈溢出时在此处提出的其他问题中描述的相同技术。例如，此处。但是，我无法使用我检索的数据。似乎缓冲区的大小比我试图检索的文件小32个字节（从计算检索的数据长度）。这可能与我的SFTP连接有关吗？或者我如何创建我的阅读流？

I have a pretty strange problem working with read streams in Node.js. I'm using SSH2 to create a sftp connection between me and a sftp server. I then try to create a read stream from the sftp stream. From the read stream's emitted 'data' event I append the data to an array. When the read stream's 'close' event occurs I make a call to Buffer.concat to create concat all the chunks of data I have retrieved into one buffer. This is the same technique described in other questions asked here at stack overflow. For example here. However, I'm not able to use the data I retrieve. It seems like the buffer is 32 bytes less in size then the file that I'm trying to retrieve (from counting the length of the data retrieved). Could this have something to do with my SFTP connection? Or how I create my read stream?

如果重要，该文件的类型为zip。当我尝试解压缩文件（在node.js中并手动）后，将其读取到缓冲区后，它无法正常工作。

If it matters, the file is of type zip. When I'm trying to unzip the file (in node.js and manually) after reading it to buffer it doesn't work.

调查后我发现：

1. 当我使用readdir时文件，文件的大小是正确的。


2. 使用FTP（JSFTP）对我的开发FTP服务器使用上述相同的技术可以正常工作。

任何建议表示赞赏！

这是我的代码：

        var Client = require('ssh2').Client;
        var m_ssh2Credentials = {
           host: config.ftpHostName,
           port: config.ftpPort,
           username: config.ftpUser,
           password: config.ftpPassword,
           readyTimeout: 20000,
           algorithms: { cipher: ["3des-cbc", "aes256-cbc", "aes192-cbc","aes128-cbc"]}
        };
        ...
        var conn = new Client();
        var dataLength = 0;
        conn.on('ready', function() {
            conn.sftp(function(err, sftp) {
                if (err) {
                    writeToErrorLog("downloadFile(): Failed to open SFTP connection.");
                } else {
                    writeToLog("downloadFile(): Opened SFTP connection.");
                }

                var streamErr = "";
                var dataLength = 0;
                var stream = sftp.createReadStream(config.ftpPath + "/" + m_fileName)
                stream.on('data', function(d){
                    data.push(d);
                    dataLength += d.length;
                });
                .on('error', function(e){
                    streamErr = e;
                })
                .on('close', function(){
                    if(streamErr) {
                        writeToErrorLog("downloadFile(): Error retrieving the file: " + streamErr);
                    } else {
                        writeToLog("downloadFile(): No error using read stream.");
                        m_fileBuffer = Buffer.concat(data, dataLength);
                        writeToLog("Data length: " + dataLength);

                        writeToLog("downloadFile(): File saved to buffer.");
                    }
                    conn.end();
                });
            })
        })
        .on('error', function(err) {
            writeToErrorLog("downloadFile(): Error connecting: " + err);
        }).connect(m_ssh2Credentials);

推荐答案

经过大量调查后我终于意识到那里在'data'事件中传输的最后几位数据有问题。根据我的理解，它似乎是读取流的实现中的一个错误。通过在SSH2库中使用更简单的函数（open，fstat，read），我能够解决这个问题。这个解决方案适合我。如果其他人遇到同样的问题，想要分享解决方案。

So after a lot of investigation I finally realized that there was something wrong with the last bits of data that was transmitted in the 'data' event. From my understanding it seems to be a bug in the implementation of the read stream. I was able to get around this problem by using the more simplistic functions (open, fstat, read) in the SSH2 library. This solution works for me. Wanted to share the solution if someone else comes across the same problem.

工作代码：

sftp.open(config.ftpPath + "/" + m_fileName, "r", function(err, fd) {
sftp.fstat(fd, function(err, stats) {
    var bufferSize = stats.size,
        chunkSize = 16384,
        buffer = new Buffer(bufferSize),
        bytesRead = 0,
        errorOccured = false;

    while (bytesRead < bufferSize && !errorOccured) {
        if ((bytesRead + chunkSize) > bufferSize) {
            chunkSize = (bufferSize - bytesRead);
        }
        sftp.read(fd, buffer, bytesRead, chunkSize, bytesRead, callbackFunc);
        bytesRead += chunkSize;
    }

    var totalBytesRead = 0;
    function callbackFunc(err, bytesRead, buf, pos) {
        if(err) {
            writeToErrorLog("downloadFile(): Error retrieving the file.");
            errorOccured = true;
            sftp.close(fd);
        }
        totalBytesRead += bytesRead;
        data.push(buf);
        if(totalBytesRead === bufferSize) {
            m_fileBuffer = Buffer.concat(data);
            writeToLog("downloadFile(): File saved to buffer.");
            sftp.close(fd);
            m_eventEmitter.emit(' downloadFile_Completed ');
        }
    }
})
});   

这篇关于使用Node.js和SSH2从SFTP服务器读取文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！