两台服务器之间具有远程rsync的管道中断 [英] broken pipe with remote rsync between two servers

问题描述

我正在尝试使用ubuntu 16.04上的bash将大型数据集(768 Gigs)从一台远程计算机传输到另一台远程计算机.我似乎遇到的问题是我使用rsync，并且计算机将传输数小时，然后在不可避免地中断连接时退出.因此，假设计算机A上的Im和远程服务器分别是计算机B和C(所有使用ubuntu 16.04的计算机).我将SSH切换到机器B并使用以下命令:

I am trying to transfer a large dataset (768 Gigs) from one remote machine to another using bash on ubuntu 16.04. The problem I appear to be having is that I use rsync and the machine will transfer for a few hours and then quit when the connection inevitably gets interrupted. So suppose Im on machine A and the remote servers are machines B and C (all machines using ubuntu 16.04). I ssh to machine B and use this command:

nohup rsync -P -r -e ssh /path/to/files/on/machine_B user@machine_C:directory &

请注意，我具有授权的密钥设置，因此机器B和C之间不需要密码

note that I have the authorized key setup so no password is required between machines B and C

几个小时后，我在nohup文件中得到以下内容:

A few hours later I get the following in the nohup file:

sending incremental filelist
file_1.bam
90,310,583,648 100%   36.44MB/s    0:39:23 (xfr#4, to-chk=5/10)
file_2.bam
79,976,321,885 100%   93.25MB/s    0:13:37 (xfr#3, to-chk=6/10)
file_3.bam
88,958,959,616  88%   12.50MB/s    0:15:28  rsync error: unexplained error (code 129) at rsync.c(632) [sender=3.1.1]
rsync: [sender] write error: Broken pipe (32)

我使用了nohup，因为即使挂断它也可以继续运行.我还没有尝试过sh -c，也没有尝试过从机器A运行命令，因为在这一点上我尝试做的都是猜测，想法将不胜感激.

I used nohup because I though it would keep running even if there was a hangup. I have not tried sh -c and I have not tried running the command from machine A because at this point whatever I try would be guesswork, ideas would be appreciated.

推荐答案

对于那些感兴趣的人，我还尝试在计算机B上使用nohup命令运行以下脚本.

for those that are interested I also tried running the following script with the nohup command on machine B.

脚本:

chomp( my @files = `ls /path/to/files/on/machineB/*` );
foreach ( @files ) { system("scp $_ user@machineC:destination/"); }

我仍然有被截断的文件.

I still got truncated files.

此刻以下命令似乎正在起作用:

at the moment the following command appears to be working:

nohup rsync -P --append -r -e ssh /path/to/files/on/machine_B user@machine_C:directory &

您只需要检查nohup文件中是否存在管道破裂错误，并在必要时重新输入命令.

you just have to check the nohup file for a broken pipe error and re-enter the command if necessary.

这篇关于两台服务器之间具有远程rsync的管道中断的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！