如何有效地将许多文件移至新服务器? [英] How can I efficiently move many files to a new server?

问题描述

我要转换托管服务提供商，需要将数百万个已上传的文件传输到新服务器.所有文件都在同一目录中.是的.您没看错. ;)

I'm switching hosting providers and need to transfer millions of uploaded files to a new server. All of the files are in the same directory. Yes. You read that correctly. ;)

过去，我已经这样做了:

In the past I've done this:

1. 压缩源服务器中的所有文件
2. scp到新服务器的zip
3. 解压缩
4. 将目录移动到适当的位置
   • 无论出于何种原因，我从第1步开始的拉链总会带来路径并要求我进行MV.

1. Zip all of the files from the source server
2. scp the zip to the new server
3. Unzip
4. Move directory to appropriate location
   • for whatever reason my zips from step 1 always bring the path along with them and require me to mv.

我上一次执行此操作大约需要4-5天，这大约是我现在的60％.

The last time I did this it took about 4-5 days to complete and that was about 60% of what I have now.

我希望有一个更好的方法.你有什么建议?

I'm hoping for a better way. What do you suggest?

文件结构被散列.像这样的东西:AAAAAAAAAA.jpg-ZZZZZZZZZZ.txt

File structure is hashed. Something like this: AAAAAAAAAA.jpg - ZZZZZZZZZZ.txt

这是我们要解决的一个想法:

Here's one idea we're tossing around:

根据3个字母的前缀将拉链拆分为成堆的迷你拉链.像这样:

Split the zips into tons of mini-zips based on 3 letter prefixes. Something like:

AAAAAAAAAA.jpg - AAAZZZZZZZ.gif => AAA.zip

理论上的优点:

• 可以加快传输速度，允许一次传输多个拉链
• 可以限制因传输失败而浪费的时间. (等待2天的转移最终失败很可怕)

理论上的缺点:

• 可能会大大降低初始zip的速度，因为该zip必须通过通配符(AAA*)查找文件，也许可以通过使用所有CPU而不是仅使用一个CPU一次运行多个zip线程来抵消.
• 复杂性?

• could slow down the initial zip considerably since the zip has to look up the files through a wildcard (AAA*), perhaps offset by running many zip threads at once, using all CPUs instead of only one.
• Complexity?

我们也考虑过rsync和scp，但担心手动传输每个文件的开销.而且由于远程服务器是空的，所以我不必担心那里已经有东西.

We've also thought about rsync and scp but worry about the expense of transferring each file manually. And since the remote server is empty I don't need to worry about what's already there.

您怎么看?你会怎么做?

What do you think? How would you do it?

(是的，我最终会将它们移动到Amazon S3，我将仅将它们运送到磁盘上，但是与此同时，我昨天需要它们！)

(Yes, I'll be moving these to Amazon S3 eventually, and I'll just ship them a disk, but in the meantime, I need them up yesterday!)

推荐答案

您实际上有多个选择，我最喜欢的是使用rsync.

You actually have multiple options, my favorite would be using rsync.

rsync [dir1] [dir2]

此命令实际上将比较目录，并且仅同步它们之间的差异.

This command will actually compare the directories, and sync only the differences between them.

有了这个，我很可能会使用以下内容

With this, I would be most likeley to use the following

rsync -z -e ssh [e-mail]:/var/www/ /var/www/

-z邮编
-e Shell命令

-z Zip
-e Shell Command

您还可以使用SFTP，通过SSH的FTP.

You could also use SFTP, FTP via SSH.

甚至是wget.

wget -rc ssh://[e-mail]:/var/www/

这篇关于如何有效地将许多文件移至新服务器?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！