使用apache commons-net FTPClient传输原始二进制文件？ [英] Transfer raw binary with apache commons-net FTPClient?

问题描述

UPDATE：已解决

我打电话给 FTPClient.setFileType（） strong>之前我登录，导致FTP服务器使用默认模式（ ASCII ），无论我设置为 。另一方面，客户端表现得好像文件类型已被正确设置。 BINARY 模式现在可以按照需要正常工作，在所有情况下都以字节为单位传输文件。我所要做的只是在wireshark中进行一些流量嗅探，然后使用netcat模拟FTP命令来查看发生了什么。为什么两天前我没有想到！？感谢大家的帮助！

我有一个xml文件，utf-16编码，我使用apache的commons-net-2.0 java库从FTP站点下载FTPClient。它支持两种传输模式： ASCII_FILE_TYPE 和 BINARY_FILE_TYPE ，区别在于 ASCII 会用适当的本地行分隔符（'\r\''或'\\\
>）替换行分隔符' - 十六进制， 0x0d0a 或者只是 0x0a ）。我的问题是：我有一个测试文件，utf-16编码，包含以下内容：

<？xml version ='1.0 'encoding ='utf-16'？>

< data>

     < blah> blah< / blah>

< / data>

下面是十六进制：

0000000：003c 003f 0078 006d 006c 0020 0076 0065 <？xml的.ve

0000010：0072 0073 0069 006f 006e 003d 0027 0031 .rsion =。'。1
0000020：002e 0030 0027 0020 0065 006e 0063 006f ... 0。'。 .enco

0000030：0064 0069 006e 0067 003d 0027 0075 0074 .ding =。'。ut
0000040：0066 002d 0031 0036 0027 003f 003e 000a .f .-。1.6。'。？。> ..

0000050：003c 0064 0061 0074 0061 003e 000a 0009。< .data> ....

0000060 ：003c 0062 006c 0061 0068 003e 0062 006c。< .blah> .bl

0000070：0061 0068 003c 002f 0062 006c 0061 0068。 ah< ./。blah

0000080：003e 000a 003c 002f 0064 0061 0074 0061。> ...< ./。data

0000090：003e 000a                                                               > ..


当我为此文件使用 ASCII 模式时，它会正确传输，字节为字节;结果有相同的md5sum。大。当我使用 BINARY 传输模式时，除了将字节从 InputStream 转换为 OutputStream ，结果是换行符（ 0x0a ）被转换为回车符+换行符对（ 0x0d0a ）。这是二进制转换后的十六进制：

0000000：003c 003f 0078 006d 006c 0020 0076 0065。<。<。x.m.l。 .ve

0000010：0072 0073 0069 006f 006e 003d 0027 0031 .rsion =。'。1
0000020：002e 0030 0027 0020 0065 006e 0063 006f ... 0。'。 .enco

0000030：0064 0069 006e 0067 003d 0027 0075 0074 .ding =。'。ut
0000040：0066 002d 0031 0036 0027 003f 003e 000d .f .-。1.6。'。？。> ..

0000050：0a00 3c00 6400 6100 7400 6100 3e00 0d0a ..< .data> ...

0000060 ：0009 003c 0062 006c 0061 0068 003e 0062 ...< .blah> .b

0000070：006c 0061 0068 003c 002f 0062 006c 0061.lah <./。bla

0000080：0068 003e 000d 0a00 3c00 2f00 6400 6100 .h。> ....< ; ./。da

0000090：7400 6100 3e00 0d0a        ;                        &NBSP ;          ta> ...

它不仅转换换行符（它不应该这样做） ），但它不尊重utf-16编码（不是我期望它知道它应该，它只是一个愚蠢的FTP管道）。如果没有进一步的处理来重新排列字节，结果是不可读的。我只是使用 ASCII 模式，但我的应用程序也将在同一个管道中移动真实的二进制数据（mp3文件和jpeg图像）。对这些二进制文件使用 BINARY 传输模式也会导致它们在其内容中注入随机 0x0d s，因为二进制数据通常包含合法的 0x0d0a 序列，所以不能安全地删除。如果我在这些文件上使用 ASCII 模式，那么巧妙的FTPClient会将这些 0x0d0a s转换为 0x0a 无论我做什么，都会使文件不一致。

我想我的问题是（是）：有人知道任何优秀的FTP库，只是将那些该死的字节从那里移动到这里，还是我将不得不破解apache commons-net-2.0并为这个简单的应用程序维护我自己的FTP客户端代码？有没有人处理这种奇怪的行为？任何建议，将不胜感激。

我检查了commons-net源代码，它看起来不像它负责怪异的行为，当使用BINARY 模式。但是从 BINARY 模式读取的 InputStream 只是一个 java.io.BufferedInptuStream 包装在套接字 InputStream 中。这些较低级别的Java流是否会执行任何奇怪的字节操作？如果他们这样做，我会感到震惊，但我不知道还有什么可以做的。

编辑1：

下面是一段代码，它模仿我正在做什么来下载文件。要编译，只需要执行

  javac -classpath /path/to/commons-net-2.0.jar Main.java $ b $为了运行，你需要目录/ tmp / ascii和/ tmp / binary来下载文件，以及一个在其中放置文件的ftp站点。该代码还需要使用适当的ftp主机，用户名和密码进行配置。我把这个文件放在测试/文件夹下的我的测试ftp站点上，并调用文件test.xml。测试文件应该至少有多行，并且是utf-16编码的（这可能不是必需的，但将有助于重新创建我的确切情况）。在打开一个新文件并输入上面提到的xml文本后，我使用了vim的：set fileencoding = utf-16 命令。最后，运行，只需要执行 
 
 
  java -cp。：/ path / to / commons-net-2.0.jar Main 
  
代码： 
 
 
修改为使用自定义FTPClient对象，链接在下面的编辑2下） 
 
 
  import java.io. *; 
 import java.util.zip.CheckedInputStream; 
 import java.util.zip.CheckedOutputStream; 
 import java.util.zip.CRC32; 
 import org.apache.commons.net.ftp。*; 
 
 public class main实现java.io.Serializable 
 {
 public static void main（String [] args）throws Exception 
 {
 Main main =新的Main（）; 
 main.doTest（）; 
} 
 
 private void doTest（）抛出异常
 {
 String host =ftp.host.com; 
 String user =user; 
 String pass =pass; 
 
 String asciiDest =/ tmp / ascii; 
字符串binaryDest =/ tmp / binary; 
 
 String remotePath =test /; 
字符串remoteFilename =test.xml; 
 
 System.out.println（TEST.XML ASCII）; 
 MyFTPClient client = createFTPClient（host，user，pass，org.apache.commons.net.ftp.FTP.ASCII_FILE_TYPE）; 
 File path = new File（/ tmp / ascii）; 
 downloadFTPFileToPath（客户端，test /，test.xml，路径）; 
 System.out.println（）; 
 
 System.out.println（TEST.XML BINARY）; 
 client = createFTPClient（host，user，pass，org.apache.commons.net.ftp.FTP.BINARY_FILE_TYPE）; 
 path = new File（/ tmp / binary）; 
 downloadFTPFileToPath（客户端，test /，test.xml，路径）; 
 System.out.println（）; 
 
 System.out.println（TEST.MP3 ASCII）; 
 client = createFTPClient（host，user，pass，org.apache.commons.net.ftp.FTP.ASCII_FILE_TYPE）; 
 path = new File（/ tmp / ascii）; 
 downloadFTPFileToPath（客户端，test /，test.mp3，路径）; 
 System.out.println（）; 
 
 System.out.println（TEST.MP3 BINARY）; 
 client = createFTPClient（host，user，pass，org.apache.commons.net.ftp.FTP.BINARY_FILE_TYPE）; 
 path = new File（/ tmp / binary）; 
 downloadFTPFileToPath（客户端，test /，test.mp3，路径）; 
} 
 
 public static File downloadFTPFileToPath（MyFTPClient ftp，String remoteFileLocation，String remoteFileName，File path）
抛出异常
 {
 //远程路径资源
字符串remoteFilePath = remoteFileLocation +/+ remoteFileName; 
 
 //创建本地结果文件对象
文件resultFile = new File（path，remoteFileName）; 
 
 //本地文件输出流
 CheckedOutputStream fout = new CheckedOutputStream（new FileOutputStream（resultFile），new CRC32（））; 
 
 //尝试从远程服务器读取数据
 if（ftp.retrieveFile（remoteFilePath，fout））{
 System.out.println（FileOut：+ fout。 。getChecksum（）的getValue（））; 
返回resultFile; 
} else {
抛出新的异常（无法完全下载文件：+ remoteFilePath）; 
 
 
 
 public static MyFTPClient createFTPClient（String url，String user，String pass，int type）
 throws异常
 {
 MyFTPClient ftp = new MyFTPClient（）; 
 ftp.connect（url）; 
 if（！ftp.setFileType（type））{
抛出新的异常（无法将ftpClient对象设置为BINARY_FILE_TYPE）; 
} 
 
 //检查连接是否成功
 int reply = ftp.getReplyCode（）; 
 if（！FTPReply.isPositiveCompletion（reply））{
 ftp.disconnect（）; 
抛出新的异常（无法正确连接到FTP）; 
 
 
 //尝试登录
 if（！ftp.login（user，pass））{
 String msg =无法登录到FTP; 
 ftp.disconnect（）; 
抛出新的异常（msg）; 
} 
 
 //成功！返回连接的MyFTPClient。 
返回ftp; 
} 
 
} 
  
编辑2：
 
 
 
好吧，我遵循 CheckedXputStream 的建议，这里是我的结果。我制作了一个名为 MyFTPClient 的apache的 FTPClient 副本，并且将 SocketInputStream 和 BufferedInputStream 在 CheckedInputStream 中使用 CRC32 校验和。此外，我将> FileOutputStream 包装到 FTPClient 中，以将输出存储在 CheckOutputStream 与 CRC32 校验和。 MyFTPClient的代码发布在这里，我修改了上述测试代码以使用此版本的FTP客户端（试图发布一个gist URL到修改后的代码，但我需要10个信誉点才能发布多个URL！）， test.xml 和 test.mp3 ，结果如下： 
 
 
  14：00：08,644 DEBUG [main，TestMain ] TEST.XML ASCII 
 14：00：08,919 DEBUG [main，MyFTPClient]套接字CRC32：2739864033 
 14：00：08,919调试[main，MyFTPClient]缓冲区CRC32：2739864033 
 14:00 ：08,954调试[main，FTPUtils] FileOut CRC32：866869773 
 
 14：00：08,955调试[main，TestMain] TEST.XML BINARY 
 14：00：09,270 DEBUG [main，MyFTPClient] Socket CRC32：2739864033 
 14：00：09,270 DEBUG [main，MyFTPClient]缓冲区CRC32：2739864033 
 14：00：09,310 DEBUG [main，FTPUtils] FileOut CRC32：2739864033 
 
 14：00：09,310 DEBUG [main，TestMain] TEST.MP3 ASCII 
 14：00：10,635 DEBUG [main，MyFTP客户端]套接字CRC32：60615183 
 14：00：10,635 DEBUG [main，MyFTPClient]缓冲区CRC32：60615183 
 14：00：10,636调试[main，FTPUtils] FileOut CRC32：2352009735 
 
 14：00：10,636 DEBUG [main，TestMain] TEST.MP3 BINARY 
 14：00：11,482 DEBUG [main，MyFTPClient] Socket CRC32：60615183 
 14：00：11,482 DEBUG [main，MyFTPClient ]缓冲区CRC32：60615183 
 14：00：11,483 DEBUG [main，FTPUtils] FileOut CRC32：60615183 
  
这使得基本上没有任何感觉，因为这里是相应文件的md5sum： 
 
 
  bf89673ee7ca819961442062eaaf9c3f ascii / test.mp3 
 7bd0e8514f1b9ce5ebab91b8daa52c4b二进制/请将test.mp3 
 ee172af5ed0204cf9546d176ae00a509原/请将test.mp3 
 
 104e14b661f3e5dbde494a54334a6dd0 ASCII /的test.xml 
 36f482a709130b01d5cddab20a28a8e8二进制/的test.xml 
 104e14b661f3e5dbde494a54334a6dd0原/test.xml 
  
我很茫然。我发誓我在这个过程的任何时候都没有对文件名/路径进行排序，并且我已经三次检查了每一步。它一定是简单的东西，但是我还没有最后的想法。为了实用性，我打算继续调用shell来完成我的FTP传输，但我打算继续研究，直到我明白到底发生了什么。我会用我的调查结果更新此主题，我将继续感谢任何人可能有的贡献。希望这对某些人有用！
解决方案
登录到ftp服务器后
  ftp.setFileType（FTP.BINARY_FILE_TYPE）; 
  
下面的行并没有解决它：
  // ftp.setFileTransferMode（org.apache.commons.net.ftp.FTP.BINARY_FILE_TYPE）; 
  
 
UPDATE: Solved


I was calling FTPClient.setFileType() before I logged in, causing the FTP server to use the default mode (ASCII) no matter what I set it to. The client, on the other hand, was behaving as though the file type had been properly set. BINARY mode is now working exactly as desired, transporting the file byte-for-byte in all cases. All I had to do was a little traffic sniffing in wireshark and then mimicing the FTP commands using netcat to see what was going on. Why didn't I think of that two days ago!? Thanks, everyone for your help!

I have an xml file, utf-16 encoded, which I am downloading from an FTP site using apache's  commons-net-2.0 java library's FTPClient. It offers support for two transfer modes: ASCII_FILE_TYPE and BINARY_FILE_TYPE, the difference being that ASCII will replace line separators with the appropriate local line separator ('\r\n' or just '\n' -- in hex, 0x0d0a or just 0x0a). My problem is this: I have a test file, utf-16 encoded, that contains the following:

<?xml version='1.0' encoding='utf-16'?>

<data>

    <blah>blah</blah>

</data>

Here's the hex:

0000000: 003c 003f 0078 006d 006c 0020 0076 0065  .<.?.x.m.l. .v.e

0000010: 0072 0073 0069 006f 006e 003d 0027 0031  .r.s.i.o.n.=.'.1

0000020: 002e 0030 0027 0020 0065 006e 0063 006f  ...0.'. .e.n.c.o

0000030: 0064 0069 006e 0067 003d 0027 0075 0074  .d.i.n.g.=.'.u.t

0000040: 0066 002d 0031 0036 0027 003f 003e 000a  .f.-.1.6.'.?.>..

0000050: 003c 0064 0061 0074 0061 003e 000a 0009  .<.d.a.t.a.>....

0000060: 003c 0062 006c 0061 0068 003e 0062 006c  .<.b.l.a.h.>.b.l

0000070: 0061 0068 003c 002f 0062 006c 0061 0068  .a.h.<./.b.l.a.h

0000080: 003e 000a 003c 002f 0064 0061 0074 0061  .>...<./.d.a.t.a

0000090: 003e 000a                                                            .>..


When I use ASCII mode for this file it transfers correctly, byte-for-byte; the result has the same md5sum. Great. When I use BINARY transfer mode, which is not supposed to do anything but shuffle bytes from an InputStream into an OutputStream, the result is that the newlines (0x0a) are converted to carriage return + newline pairs (0x0d0a). Here's the hex after binary transfer:


0000000: 003c 003f 0078 006d 006c 0020 0076 0065  .<.?.x.m.l. .v.e

0000010: 0072 0073 0069 006f 006e 003d 0027 0031  .r.s.i.o.n.=.'.1

0000020: 002e 0030 0027 0020 0065 006e 0063 006f  ...0.'. .e.n.c.o

0000030: 0064 0069 006e 0067 003d 0027 0075 0074  .d.i.n.g.=.'.u.t

0000040: 0066 002d 0031 0036 0027 003f 003e 000d  .f.-.1.6.'.?.>..

0000050: 0a00 3c00 6400 6100 7400 6100 3e00 0d0a  ..<.d.a.t.a.>...

0000060: 0009 003c 0062 006c 0061 0068 003e 0062  ...<.b.l.a.h.>.b

0000070: 006c 0061 0068 003c 002f 0062 006c 0061  .l.a.h.<./.b.l.a

0000080: 0068 003e 000d 0a00 3c00 2f00 6400 6100  .h.>....<./.d.a.

0000090: 7400 6100 3e00 0d0a                                        t.a.>...


Not only does it convert the newline characters (which it shouldn't), but it doesn't respect the utf-16 encoding (not that I would expect it to know that it should, it's just a dumb FTP pipe). The result is unreadable without further processing to realign the bytes. I would just use ASCII mode, but my application will also be moving real binary data (mp3 files and jpeg images) across the same pipe. Using the BINARY transfer mode on these binary files also causes them to have random 0x0ds injected into their contents, which can't safely be removed since the binary data often contains legitimate 0x0d0a sequences. If I use ASCII mode on these files, then the "clever" FTPClient converts these 0x0d0as into 0x0a leaving the file inconsistent no matter what I do.

I guess my question(s) is(are): does anyone know of any good FTP libraries for java that just move the damned bytes from there to here, or am I going to have to hack up apache commons-net-2.0 and maintain my own FTP client code just for this simple application? Has anyone else dealt with this bizarre behavior? Any suggestions would be appreciated.

I checked out the commons-net source code and it doesn't look like it's responsible for the weird behavior when BINARY mode is used. But the InputStream it's reading from in BINARY mode is just a java.io.BufferedInptuStream wrapped around a socket InputStream. Do these lower level java streams ever do any weird byte-manipulation? I would be shocked if they did, but I don't see what else could be going on here.

EDIT 1:

Here's a minimal piece of code that mimics what I'm doing to download the file. To compile, just do
javac -classpath /path/to/commons-net-2.0.jar Main.java
To run, you'll need directories /tmp/ascii and /tmp/binary for the file to download to, as well as an ftp site set up with the file sitting in it. The code will also need to be configured with the appropriate ftp host, username and password. I put the file on my testing ftp site under the test/ folder and called the file test.xml. The test file should at least have more than one line, and be utf-16 encoded (this may not be necessary, but will help to recreate my exact situation). I used vim's :set fileencoding=utf-16 command after opening a new file and entered the xml text referenced above. Finally, to run, just do
java -cp .:/path/to/commons-net-2.0.jar Main
Code:

(NOTE: this code modified to use custom FTPClient object, linked below under "EDIT 2")
import java.io.*;
import java.util.zip.CheckedInputStream;
import java.util.zip.CheckedOutputStream;
import java.util.zip.CRC32;
import org.apache.commons.net.ftp.*;

public class Main implements java.io.Serializable
{
    public static void main(String[] args) throws Exception
    {
        Main main = new Main();
        main.doTest();
    }

    private void doTest() throws Exception
    {
        String host = "ftp.host.com";
        String user = "user";
        String pass = "pass";

        String asciiDest = "/tmp/ascii";
        String binaryDest = "/tmp/binary";

        String remotePath = "test/";
        String remoteFilename = "test.xml";

        System.out.println("TEST.XML ASCII");
        MyFTPClient client = createFTPClient(host, user, pass, org.apache.commons.net.ftp.FTP.ASCII_FILE_TYPE);
        File path = new File("/tmp/ascii");
        downloadFTPFileToPath(client, "test/", "test.xml", path);
        System.out.println("");

        System.out.println("TEST.XML BINARY");
        client = createFTPClient(host, user, pass, org.apache.commons.net.ftp.FTP.BINARY_FILE_TYPE);
        path = new File("/tmp/binary");
        downloadFTPFileToPath(client, "test/", "test.xml", path);
        System.out.println("");

        System.out.println("TEST.MP3 ASCII");
        client = createFTPClient(host, user, pass, org.apache.commons.net.ftp.FTP.ASCII_FILE_TYPE);
        path = new File("/tmp/ascii");
        downloadFTPFileToPath(client, "test/", "test.mp3", path);
        System.out.println("");

        System.out.println("TEST.MP3 BINARY");
        client = createFTPClient(host, user, pass, org.apache.commons.net.ftp.FTP.BINARY_FILE_TYPE);
        path = new File("/tmp/binary");
        downloadFTPFileToPath(client, "test/", "test.mp3", path);
    }

    public static File downloadFTPFileToPath(MyFTPClient ftp, String remoteFileLocation, String remoteFileName, File path)
        throws Exception
    {
        // path to remote resource
        String remoteFilePath = remoteFileLocation + "/" + remoteFileName;

        // create local result file object
        File resultFile = new File(path, remoteFileName);

        // local file output stream
        CheckedOutputStream fout = new CheckedOutputStream(new FileOutputStream(resultFile), new CRC32());

        // try to read data from remote server
        if (ftp.retrieveFile(remoteFilePath, fout)) {
            System.out.println("FileOut: " + fout.getChecksum().getValue());
            return resultFile;
        } else {
            throw new Exception("Failed to download file completely: " + remoteFilePath);
        }
    }

    public static MyFTPClient createFTPClient(String url, String user, String pass, int type)
        throws Exception
    {
        MyFTPClient ftp = new MyFTPClient();
        ftp.connect(url);
        if (!ftp.setFileType( type )) {
            throw new Exception("Failed to set ftpClient object to BINARY_FILE_TYPE");
        }

        // check for successful connection
        int reply = ftp.getReplyCode();
        if (!FTPReply.isPositiveCompletion(reply)) {
            ftp.disconnect();
            throw new Exception("Failed to connect properly to FTP");
        }

        // attempt login
        if (!ftp.login(user, pass)) {
            String msg = "Failed to login to FTP";
            ftp.disconnect();
            throw new Exception(msg);
        }

        // success! return connected MyFTPClient.
        return ftp;
    }

}
EDIT 2:

Okay I followed the CheckedXputStream advice and here are my results. I made a copy of apache's FTPClient called MyFTPClient, and I wrapped both the SocketInputStream and the BufferedInputStream in a CheckedInputStream using CRC32 checksums. Furthermore, I wrapped the FileOutputStream that I give to FTPClient to store the output in a CheckOutputStream with CRC32 checksum. The code for MyFTPClient is posted here and I've modified the above test code to use this version of the FTPClient (tried to post a gist URL to the modified code, but I need 10 reputation points to post more than one URL!), test.xml and test.mp3 and the results were thus:
14:00:08,644 DEBUG [main,TestMain] TEST.XML ASCII
14:00:08,919 DEBUG [main,MyFTPClient] Socket CRC32: 2739864033
14:00:08,919 DEBUG [main,MyFTPClient] Buffer CRC32: 2739864033
14:00:08,954 DEBUG [main,FTPUtils] FileOut CRC32: 866869773

14:00:08,955 DEBUG [main,TestMain] TEST.XML BINARY
14:00:09,270 DEBUG [main,MyFTPClient] Socket CRC32: 2739864033
14:00:09,270 DEBUG [main,MyFTPClient] Buffer CRC32: 2739864033
14:00:09,310 DEBUG [main,FTPUtils] FileOut CRC32: 2739864033

14:00:09,310 DEBUG [main,TestMain] TEST.MP3 ASCII
14:00:10,635 DEBUG [main,MyFTPClient] Socket CRC32: 60615183
14:00:10,635 DEBUG [main,MyFTPClient] Buffer CRC32: 60615183
14:00:10,636 DEBUG [main,FTPUtils] FileOut CRC32: 2352009735

14:00:10,636 DEBUG [main,TestMain] TEST.MP3 BINARY
14:00:11,482 DEBUG [main,MyFTPClient] Socket CRC32: 60615183
14:00:11,482 DEBUG [main,MyFTPClient] Buffer CRC32: 60615183
14:00:11,483 DEBUG [main,FTPUtils] FileOut CRC32: 60615183
This makes, basically zero sense whatsoever because here are the md5sums of the corresponsing files:
bf89673ee7ca819961442062eaaf9c3f  ascii/test.mp3
7bd0e8514f1b9ce5ebab91b8daa52c4b  binary/test.mp3
ee172af5ed0204cf9546d176ae00a509  original/test.mp3

104e14b661f3e5dbde494a54334a6dd0  ascii/test.xml
36f482a709130b01d5cddab20a28a8e8  binary/test.xml
104e14b661f3e5dbde494a54334a6dd0  original/test.xml
I'm at a loss. I swear I haven't permuted the filenames/paths at any point in this process, and I've triple-checked every step. It must be something simple, but I haven't the foggiest idea where to look next. In the interest of practicality I'm going to proceed by calling out to the shell to do my FTP transfers, but I intend to pursue this until I understand what the hell is going on. I'll update this thread with my findings, and I'll continue to appreciate any contributions anyone may have. Hopefully this will be useful to someone at some point!
解决方案
After login to the ftp server
ftp.setFileType(FTP.BINARY_FILE_TYPE);
The line below doesn't solve it:
//ftp.setFileTransferMode(org.apache.commons.net.ftp.FTP.BINARY_FILE_TYPE);


                        
这篇关于使用apache commons-net FTPClient传输原始二进制文件？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！