FTP：复制，检查完整性和删除 [英] FTP: copy, check integrity and delete

问题描述

我正在寻找一种连接到远程服务器的方式，使用 ftp 或 lftp 并确保以下内容步骤：
$ b

1. 将文件从FTP服务器复制到本地计算机。




2. 检查下载的文件是否正常（即md5checksum）。

如果下载没问题，请从FTP服务器上删除下载的文件。




这个程序将每天从我的本地机器执行。什么是最好的选择来做到这一点？是否有一个工具可以抽象出所有3个步骤？





我在客户端和服务器上运行Linux。





更新：此外，我还有一个文本文件，其中包含FTP服务器上的文件与其MD5sum之间的关联。它们是在FTP服务器端进行计算的。首先，确保远程服务器支持校验和计算。许多人不知道。我相信甚至没有标准的FTP命令来计算远程文件的校验和。有许多提案，并有许多专有解决方案。





最新提议是：

https://tools.ietf.org/html/draft-bryan-ftpext-hash-02





因此，即使您的服务器支持校验和计算，您也必须找到支持相同命令的客户端。





可用于计算校验和的命令为： XSHA1 ， XSHA256 ， XSHA512 ， XMD5 ， MD5 ， XCRC 和 HASH 。





您可以使用WinSCP进行测试。 WinSCP支持前面提到的所有命令。测试其校验和计算函数或 checksum 脚本命令。如果它们能够工作，请启用日志记录并检查WinSCP对您的服务器使用什么命令和语法。


$ b

ftp （无论是Windows还是* nix版本）也没有 lftp 支持校验和计算，只允许自动验证下载的文件。





我甚至不知道任何其他可自动验证下载文件的客户端。





您可以在一些功能丰富的客户端的帮助下编写脚本。





---





在OP指定他/她在Linux上之前，我已经写了这个答案。我保留Windows解决方案以防别人帮助。





在Windows上，您可以使用使用WinSCP .NET程序集的PowerShell 。


$ b

  param（
 $ sessionUrl =ftp：// username：[e-mail]/，
 [Parameter（Mandatory）] 
 $ localPath， 
 [参数（必须）] 
 $ remotePath，
 [Switch] 
 $暂停= $ False 
）
 
尝试
 
＃加载WinSCP .NET程序集
 Add-Type -Path（加入路径$ PSScriptRootWinSCPnet.dll）
 
＃安装会话选项
 $ sessionOptions = New-Object WinSCP.SessionOptions 
 $ sessionOptions.ParseUrl（$ sessionUrl）; 
 
 $ session = New-Object WinSCP.Session 
 
尝试
 {
＃Connect 
 $ session.Open（$ sessionOptions）
 
写主机将$ remotePath下载到$ localPath ...
 $ session.GetFiles（$ remotePath，$ localPath）.Check（）; 
 
＃计算远程文件校验和
 $ buf = $ session.CalculateFileChecksum（sha-1，$ remotePath）
 $ remoteChecksum = [BitConverter] :: ToString（$ buf ）
写主机远程文件校验和：$ remoteChecksum
 
＃计算本地文件校验和
 $ sha1 = [System.Security.Cryptography.SHA1] :: Create（） 
 $ localStream = [System.IO.File] :: OpenRead（$ localPath）
 $ localChecksum = [BitConverter] :: ToString（$ sha1.ComputeHash（$ localStream））
 Write-主机下载文件校验和：$ localChecksum
 
＃比较cheksums 
 if（$ localChecksum -eq $ remoteChecksum）
 {
写入主机匹配，删除远程文件
 
 $ session.RemoveFiles（$ remotePath）.Check（）; 
 $ result = 0 
} 
 else 
 {
写主机不匹配
 $ result = 1 
} 
 
 finally 
 
＃断开连接，清除
 $ session.Dispose（）
} 
} 
 catch [例外] 
 {
写主机错误：$（$ _。Exception.Message）
 $ result = 1 
} 
 
＃暂停if  - 暂停开关被使用
如果（$暂停）
 {
写主机按任意键退出...
 [System.Console] :: ReadKey（）| Out-Null 
} 
 
退出$ result 
  

您可以像下面这样运行它：

  powershell -file checksum.ps1 -remotePath ./file.dat -localPath C：\path\ file.dat 
  

这部分基于WinSCP示例通过SFTP / FTP协议验证远程文件对本地文件的校验和





（我是WinSCP的作者）




I am looking for a way to connect to a remote server with ftp or lftp and make sure the following steps:

1. Copy files from FTP server to my local machine.

2. Check if the downloaded files are fine (i.e. md5checksum).

3. If the download was fine then delete the downloaded files from the FTP server.

This routine will be executed each day from my local machine. What would be the best option to do this? Is there a tool that makes abstraction of all the 3 steps ?

I am running Linux on both client and server machines.

Update: Additionally, I have also a text file that contains the association between the files on the FTPserver and their MD5sum. They were computed at the FTP server side.

解决方案

First, make sure your remote server supports the checksum calculation at all. Many do not. I believe there's even no standard FTP command to calculate a checksum of a remote file. There were many proposals and there are many proprietary solutions.

The latest proposal is:
https://tools.ietf.org/html/draft-bryan-ftpext-hash-02

So even if your server supports checksum calculation, you have to find a client that supports the same command.

Some of the commands that can be used to calculate checksum are: XSHA1, XSHA256, XSHA512, XMD5, MD5, XCRC and HASH.

You can test that with WinSCP. The WinSCP supports all the previously mentioned commands. Test its checksum calculation function or the checksum scripting command. If they work, enable logging and check, what command and what syntax WinSCP uses against your server.

---

Neither the ftp (neither Windows nor *nix version) nor the lftp support checksum calculation, let only automatic verification of downloaded file.

I'm not even aware of any other client that can automatically verify downloaded file.

You can definitely script it with a help of some feature-rich client.

---

I've wrote this answer before OP specified that he/she is on Linux. I'm keeping the Windows solution in case it helps someone else.

On Windows, you could script it with PowerShell using WinSCP .NET assembly.

param (
    $sessionUrl = "ftp://username:[e-mail]/",
    [Parameter(Mandatory)]
    $localPath,
    [Parameter(Mandatory)]
    $remotePath,
    [Switch]
    $pause = $False
)

try
{
    # Load WinSCP .NET assembly
    Add-Type -Path (Join-Path $PSScriptRoot "WinSCPnet.dll")

    # Setup session options
    $sessionOptions = New-Object WinSCP.SessionOptions
    $sessionOptions.ParseUrl($sessionUrl);

    $session = New-Object WinSCP.Session

    try
    {
        # Connect
        $session.Open($sessionOptions)

        Write-Host "Downloading $remotePath to $localPath..."
        $session.GetFiles($remotePath, $localPath).Check();

        # Calculate remote file checksum
        $buf = $session.CalculateFileChecksum("sha-1", $remotePath)
        $remoteChecksum = [BitConverter]::ToString($buf)
        Write-Host "Remote file checksum:     $remoteChecksum"

        # Calculate local file checksum
        $sha1 = [System.Security.Cryptography.SHA1]::Create()
        $localStream = [System.IO.File]::OpenRead($localPath)
        $localChecksum = [BitConverter]::ToString($sha1.ComputeHash($localStream))
        Write-Host "Downloaded file checksum: $localChecksum"

        # Compare cheksums
        if ($localChecksum -eq $remoteChecksum)
        {
            Write-Host "Match, deleting remote file"

            $session.RemoveFiles($remotePath).Check();
            $result = 0
        }
        else
        {
            Write-Host "Does NOT match"
            $result = 1
        }
    }
    finally
    {
        # Disconnect, clean up
        $session.Dispose()
    }
}
catch [Exception]
{
    Write-Host "Error: $($_.Exception.Message)"
    $result = 1
}

# Pause if -pause switch was used
if ($pause)
{
    Write-Host "Press any key to exit..."
    [System.Console]::ReadKey() | Out-Null
}

exit $result

You can run it like:

powershell -file checksum.ps1 -remotePath ./file.dat -localPath C:\path\file.dat

This is partially based on WinSCP example for Verifying checksum of a remote file against a local file over SFTP/FTP protocol.

(I'm the author on WinSCP)

这篇关于FTP：复制，检查完整性和删除的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！