rsync，'uid / gid不可能设置'情况导致未来硬链接失败，如何解决？ [英] rsync, 'uid/gid impossible to set' cases cause future hard link failure, how to fix?
问题描述
我使用rsync版本3.1.1协议版本31与cygwin版本2.3.1（0.291 / 5/3）。
我相信我需要chown所有我的Win7文件，具有未知的uid / gid的，rsync将复制，第一个rsync之前运行这些文件。
在未知的uid / gid的rsync生成'uid / gid不可能设置'消息，并且复制文件，但是不能在以后的增量备份中硬链接。
我尝试了很多rsync选项来解决这个问题，但没有成功。
？
我相信的原因是因为当我有rsync在我的增量备份创建硬链接（与适当的--link-dest =目标），只有文件某个用户和组已创建其硬链接。我有一个硬盘驱动器与600,000文件，将不允许硬链接创建后，他们通过rsync传输。所有我的其他驱动器有许多文件，这种行为也。这使得硬连接的增量备份从该驱动器无效，并且与其他驱动器不太对。
这里是Win7上的各种用户和组状态的示例驱动器：
ls -al
共828
drwxrwx --- + 1未知+ Group 0 Dec 7 21:33。
dr-xrwxr-x + 1未知+用户未知+组0 Dec 7 20:53 ..
-rwxrwx --- + 1管理员无773985 2012年1月1日1.txt
-rwxrwx --- + 1未知+用户未知+组27936 2009年3月12日DATtoKML.kml
txt文件通过最近的拖放到Windows文件资源管理器中。 DATtoKML.kml文件是一个有问题的600,000个文件的示例。
这里是目标上的文件在rsynced后的样子（使用--link- dest设置为以前的rsynced位置）：
ls -al
total 792
drwxrwxrwx 2 root root 4096 Dec 7 21:33。
drwxrwxrwx 4 root root 4096 Dec 7 21:34 ..
-rwxrwxrwx 2 544 197121 773985 Jan 1 2012 1.txt
-rwxrwxrwx 1 root root 27936 Mar 12 2009 DATtoKML.kml
正如你所看到的，DATtoKML.kml没有硬链接，永远会在每个增量rsync 。
此外，这些非硬链接文件在rsync处理时从rsync获取消息。消息是uid / gid的不可能设置。这些不是致命错误，因为文件复制，但似乎没有他们的用户和组设置，所以他们是硬链接。
这里是两个rsync的结果上述列表：
Dale @ Uva1〜
$ rsync -av --chmod = o = rwx / cygdrive / e / DATtoKML / root@192.168.0.11：/ DataVolume / shares / DalesDesktop / e + DATtoKML / 2015-12-07-FIRST
发送增量文件列表
已创建目录/ DataVolume / shares / DalesDesktop / e + DATtoKML / 2015-12-07-FIRST
./
uid 4294967295（-1）不能在/ DataVolume / shares / DalesDesktop / e + DATtoKML / 2015-12-07-FIRST上设置/。
gid 4294967295（-1）不能在/ DataVolume / shares / DalesDesktop / e + DATtoKML / 2015-12-07-FIRST /上设置。
1.txt
DATtoKML.kml
uid 4294967295（-1）不能在/ DataVolume / shares / DalesDesktop / e + DATtoKML / 2015-12-07-FIRST /上设置。 DATtoKML.kml.eR2hUv
gid 4294967295（-1）不能在/DataVolume/shares/DalesDesktop/e+DATtoKML/2015-12-07-FIRST/.DATtoKML.kml.eR2hUv上设置
发送802,347字节接收628字节1,605,950.00字节/秒
总大小是801,921加速是1.00
rsync错误：一些文件/ attrs没有转移（见前面的错误）（代码23） at main.c（1165）[sender = 3.1.1]
Dale @ Uva1〜
$ rsync -av --chmod = o = rwx --link-dest = / DataVolume / share / DalesDesktop / e + DATtoKML / 2015-12-07-FIRST / cygdrive / e / DATtoKML / root@192.168.0.11：/ DataVolume / shares / DalesDesktop / e + DATtoKML / 2015-12-07
发送增量文件列表
创建目录/ DataVolume / shares / DalesDesktop / e + DATtoKML / 2015-12-07
./
uid 4294967295（-1）不能在/ DataVolume / share / DalesDesktop / e + DATtoKML / 2015-12-07 /。
gid 4294967295（-1）不能在/ DataVolume / shares / DalesDesktop / e + DATtoKML / 2015-12-07 /上设置。
uid 4294967295（-1）不能在/DataVolume/shares/DalesDesktop/e+DATtoKML/2015-12-07/.DATtoKML.kml.1lp4od上设置
gid 4294967295（-1）是不可能设置在/DataVolume/shares/DalesDesktop/e+DATtoKML/2015-12-07/.DATtoKML.kml.1lp4od
DATtoKML.kml
发送155字节接收559字节476.00字节/秒
总大小是801,921加速是1,123.14
rsync错误：一些文件/ attr没有转移（见前面的错误）（代码23）在main.c（1165）[sender = 3.1 .1]
Dale @ Uva1〜
$
远的唯一的事情似乎工作是chown和chgrp DATtoKML.kml和文件父目录在Win7驱动器之前，我rsync他们。这对我来说似乎不切实际，不是那么难以在驱动器上几乎所有的坏，但其他驱动器有这些类型的错误分散在成千上万的目录。
我认为rsync需要这样的选项来处理'uid / gid不可能设置'案例：
uid-error = 544 --gid-error = 197121
任何人都想尝试编译一个rsync版本使用该选项？
编辑：看起来像所有的文件和目录有争议的uid和/或gid 4294967295.作为一个临时解决方案，我要写一个bash脚本，它在每个win7驱动器上找到（但只包括将被传输的文件，考虑我的rsync排除文件），并将任何坏的uid变为544和坏gids为197121.使用544和197121，因为他们是用户和
这样，当我开始看到这些uid / gid问题出现在我的rsync日志中时，我可以再次运行脚本。 / p>
希望未来会有一个真正的答案这个问题。
使用--usermap和--groupmap rsync 选项（这需要源环境和目标环境都使用3.1.0或更高版本）。在我的情况下，这将不工作，因为我的目的地（WD MyCloud）使用3.0.9。
或
使用cygwin下的gcc更改 rsync 源代码（真的不是那么难用一些C知识）。
I am using rsync version 3.1.1 protocol version 31 with cygwin version 2.3.1(0.291/5/3).
I believe that I need to chown all my Win7 files, with unknown uid/gid's, which rsync would copy, BEFORE the first rsync is run on these files.
On unknown uid/gid's rsync generates 'uid/gid impossible to set' messages and does copy the files but they can NOT be then hard linked in future incremental backups.
I have tried a lot of rsync options to solve this with no success.
How do I do this?
The reason I believe that is because when I have rsync create hard links on my incremental backups (with the appropriate --link-dest=destination), only files with a certain user and group have their hard links created. I have one hard drive with 600,000 files that won't allow hard links to be created after they are transferred by rsync. All my other drives have numerous files that behave this way too. This makes incremental backups with hard links useless from that drive and 'not quite right' with the other drives.
Here is an example of various user and group status on that Win7 drive:
ls -al
total 828
drwxrwx---+ 1 Unknown+User Unknown+Group 0 Dec 7 21:33 .
dr-xrwxr-x+ 1 Unknown+User Unknown+Group 0 Dec 7 20:53 ..
-rwxrwx---+ 1 Administrators None 773985 Jan 1 2012 1.txt
-rwxrwx---+ 1 Unknown+User Unknown+Group 27936 Mar 12 2009 DATtoKML.kml
The 1.txt file was put there through a recent drag and drop in windows file explorer. The DATtoKML.kml file is an example of the 600,000 files that are a problem. The 1.txt file has no problem being rsynced and hard linked after that.
Here is what the files look like on the destination after being rsynced (with --link-dest set to a previous rsynced location) there:
ls -al
total 792
drwxrwxrwx 2 root root 4096 Dec 7 21:33 .
drwxrwxrwx 4 root root 4096 Dec 7 21:34 ..
-rwxrwxrwx 2 544 197121 773985 Jan 1 2012 1.txt
-rwxrwxrwx 1 root root 27936 Mar 12 2009 DATtoKML.kml
As you can see, DATtoKML.kml did NOT hard link and will forever get transferred in full on every incremental rsync.
Also, these non-hardlinkable files get messages from rsync when they are worked on by rsync. The messages are that the 'uid/gid's are impossible to set'. These are not fatal errors because the files copy, but don't seem to have their users and groups set so they are hardlinkable.
Here are the two rsync's that resulted in the above listing:
Dale@Uva1 ~
$ rsync -av --chmod=o=rwx /cygdrive/e/DATtoKML/ root@192.168.0.11:/DataVolume/shares/DalesDesktop/e+DATtoKML/2015-12-07-FIRST
sending incremental file list
created directory /DataVolume/shares/DalesDesktop/e+DATtoKML/2015-12-07-FIRST
./
uid 4294967295 (-1) is impossible to set on "/DataVolume/shares/DalesDesktop/e+DATtoKML/2015-12-07-FIRST/."
gid 4294967295 (-1) is impossible to set on "/DataVolume/shares/DalesDesktop/e+DATtoKML/2015-12-07-FIRST/."
1.txt
DATtoKML.kml
uid 4294967295 (-1) is impossible to set on "/DataVolume/shares/DalesDesktop/e+DATtoKML/2015-12-07-FIRST/.DATtoKML.kml.eR2hUv"
gid 4294967295 (-1) is impossible to set on "/DataVolume/shares/DalesDesktop/e+DATtoKML/2015-12-07-FIRST/.DATtoKML.kml.eR2hUv"
sent 802,347 bytes received 628 bytes 1,605,950.00 bytes/sec
total size is 801,921 speedup is 1.00
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1165) [sender=3.1.1]
Dale@Uva1 ~
$ rsync -av --chmod=o=rwx --link-dest=/DataVolume/shares/DalesDesktop/e+DATtoKML/2015-12-07-FIRST /cygdrive/e/DATtoKML/ root@192.168.0.11:/DataVolume/shares/DalesDesktop/e+DATtoKML/2015-12-07
sending incremental file list
created directory /DataVolume/shares/DalesDesktop/e+DATtoKML/2015-12-07
./
uid 4294967295 (-1) is impossible to set on "/DataVolume/shares/DalesDesktop/e+DATtoKML/2015-12-07/."
gid 4294967295 (-1) is impossible to set on "/DataVolume/shares/DalesDesktop/e+DATtoKML/2015-12-07/."
uid 4294967295 (-1) is impossible to set on "/DataVolume/shares/DalesDesktop/e+DATtoKML/2015-12-07/.DATtoKML.kml.1lp4od"
gid 4294967295 (-1) is impossible to set on "/DataVolume/shares/DalesDesktop/e+DATtoKML/2015-12-07/.DATtoKML.kml.1lp4od"
DATtoKML.kml
sent 155 bytes received 559 bytes 476.00 bytes/sec
total size is 801,921 speedup is 1,123.14
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1165) [sender=3.1.1]
Dale@Uva1 ~
$
So far the only thing that seems to work is to chown and chgrp DATtoKML.kml and the files parent directory on the Win7 drive before I ever rsync them. This does not seem practical to me, not so hard to do on the drive where almost all of them are bad, but the other drive has these type of error scattered across thousands of directories.
I think rsync needs an option like this to handle 'uid/gid impossible to set' cases:
--uid-error=544 --gid-error=197121
Or, it should just use appropriate user and group (which it should know from the user that is rsyncing) for future hardlinkability whenever 'uid/gid impossible to set' arises.
Anyone want to try compiling an rsync version with that option?
EDIT: It looks like all the files and directories at issue have a uid and/or gid of 4294967295. As a temporary solution I am going to write a bash script which does find on each win7 drive (but only include files that will be transferred considering my rsync exclude file) and change any bad uids to 544 and bad gids to 197121. The 544 and 197121 are used because they are the user and group of the user that runs the rsync command.
That way, when I start seeing these uid/gid issues surfacing in my rsync logs I can run the script again.
Hopefully there will be a real answer to this question in the future.
Use the --usermap and --groupmap rsync options (this requires that both source and dest environments use version 3.1.0 or higher). In my case this won't work since my destination (WD MyCloud) uses 3.0.9.
OR
Change the rsync source code with gcc running under cygwin (really not that hard with some C knowledge).
这篇关于rsync，'uid / gid不可能设置'情况导致未来硬链接失败，如何解决？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
