使用bittorrent协议夜间分布和CI构建 [英] Using the bittorrent protocol to distribute nightly and CI builds

问题描述

这个问题从我昨天题为使用git分发夜间构建。

在上面的问题的答案中，很明显git不适合我的需要，使用BitTorrent进行检查。

b

每天早上需要向70岁以上的人分发夜间版本，希望使用 git BitTorrent 来平衡传输。

长版本

如果您已阅读我的上一个问题

每天早上，我们需要将70岁以上的演员（艺术家，测试人员，程序员，制作人员等）到目前为止，我们已经将构建复制到服务器，并且已经写了一个同步程序（使用下面的Robocopy）。即使设置镜像，传输速度也不可接受地慢，其中在高峰时间（非高峰时间大致为15分钟）同步需要一个小时或更长时间，这指向硬件I / O瓶颈和可能的网络带宽。

$ b

我到目前为止所知道的

/ p>

• 我在维基百科上发现了关于 BitTorrent协议这是一个有趣的阅读（我以前只知道如何洪流的工作原理的基础）。此外，还发现此 StackOverflow answer




• 我还发现了 MonoTorrent C＃Library （ GitHub Source ），我可以使用写我们自己的跟踪和客户端。我们不能使用现成的跟踪器或客户端（例如uTorrent）。

问题

在我的初始设计中，我有我们的构建系统创建一个.torrent文件和添加到跟踪器。我会使用我们现有的构建镜像超种子。

使用这个设计，我需要为每个新版本创建一个新的.torrent 文件吗？换句话说，是否有可能创建一个滚动.torrent 其中如果构建的内容只有改变20％，所有需要下载到 get ？

...其实。在写上述问题时，我认为我需要创建新文件，但是我可以下载到用户计算机上的相同位置，哈希将自动确定我已经有了。它是否正确？

回应评论

1. 对于完全新的同步，整个构建（包括：PS3和X360的游戏，源代码，本地化数据和光盘映像）〜37,000个文件，并在50GB以下 。随着生产的继续，这将增加。这个同步花费29分钟完成，当时只有2个其他同步发生，这是低峰，如果你认为在上午9点，我们将有50多个人想要获得最新。




2. 我们调查了IT部门的磁盘I / O和网络带宽;结论是网络存储已饱和。我们还将统计数据记录到同步数据库中，这些记录甚至显示了少数用户，我们获得了不可接受的传输速率。




3. 对于不使用现成的客户端，在用户上安装 uTorrent 因为其他项目可以使用该程序轻松下载。我们还想要一个自定义工作流程，以确定你想获得哪个构建（例如，只有PS3或X360取决于你的桌面上有什么DEVKIT），并有新的构建可用的通知等。使用MonoTorrent创建客户端不是部分

解决方案

你需要创建一个新的.torrent，答案是：是。

然而，根据你的数据的布局，可能可以做一些简单的半增量更新。

如果您分发的数据是单个文件的大集合，每个构建一些文件可能已更改您可以简单地创建一个新的.torrent文件，并让所有客户端下载到与旧的同一位置（就像你的建议）。客户端首先检查磁盘上已存在的文件，更新已更改的文件并下载新文件。主要缺点是删除的文件实际上不会在客户端被删除。

如果你正在写自己的客户端，无论如何，删除文件系统上的文件不是在.torrent文件是一个相当简单的步骤，可以单独完成。

这不工作，如果你分发一个图像文件，因为保持相同的位版本可能已移动，因此产生不同的散列。

我不一定建议使用超级种子。根据您使用的超级种子实现的严格程度，它可能实际上损害传输速率。请记住，超级种子的目的是最小化从种子发送的字节数，而不是最大化传输速率。

此外，要创建一个torrent和哈希表，如果你的客户端正常运行（即使用罕见的第一）检查一个50 GiB洪流给驱动器上的很多负载，你可能想要基准的bittorrent实现，你使用这个，以确保它的性能足够。在50 GiB时，不同实现之间的差异可能很大。

This questions continues from what I learnt from my question yesterday titled using git to distribute nightly builds.

In the answers to the above questions it was clear that git would not suit my needs and was encouraged to re-examine using BitTorrent.

---

Short Version

Need to distribute nightly builds to 70+ people each morning, would like to use git BitTorrent to load balance the transfer.

Long Version

NB. You can skip the below paragraph if you have read my previous question.

Each morning we need to distribute our nightly build to the studio of 70+ people (artists, testers, programmers, production etc). Up until now we have copied the build to a server and have written a sync program that fetches it (using Robocopy underneath); even with setting up mirrors the transfer speed is unacceptably slow with it taking up-to an hour or longer to sync at peak times (off-peak times are roughly 15 minutes) which points to being hardware I/O bottleneck and possibly network bandwidth.

What I know so far

What I have found so far:

• I have found the excellent entry on Wikipedia about the BitTorrent protocol which was an interesting read (I had only previously known the basics of how torrents worked). Also found this StackOverflow answer on the BITFIELD exchange that happens after the client-server handshake.

• I have also found the MonoTorrent C# Library (GitHub Source) that I can use to write our own tracker and client. We cannot use off the shelf trackers or clients (e.g. uTorrent).

Questions

In my initial design, I have our build system creating a .torrent file and adding it to the tracker. I would super-seed the torrent using our existing mirrors of the build.

Using this design, would I need to create a new .torrent file for each new build? In other words, would it be possible to create a "rolling" .torrent where if the content of the build has only change 20% that is all that needs to be downloaded to get latest?

... Actually. In writing the above question, I think that I would need to create new file however I would be able download to the same location on the users machine and the hash will automatically determine what I already have. Is this correct?

In response to comments

1. For completely fresh sync the entire build (including: the game, source code, localized data, and disc images for PS3 and X360) ~37,000 files and coming in just under 50GB. This is going to increase as production continues. This sync took 29 minutes to complete at time when there is was only 2 other syncs happening, which low-peak if you consider that at 9am we would have 50+ people wanting to get latest.

2. We have investigated the disk I/O and network bandwidth with the IT dept; the conclusion was that the network storage was being saturated. We are also recording statistics to a database of syncs, these records show even with handful of users we are getting unacceptable transfer rates.

3. In regard not using off-the-shelf clients, it is a legal concern with having an application like uTorrent installed on users machines given that other items can be easily downloaded using that program. We also want to have a custom workflow for determining which build you want to get (e.g. only PS3 or X360 depending on what DEVKIT you have on your desk) and have notifications of new builds available etc. Creating a client using MonoTorrent is not the part that I'm concerned about.

解决方案

To the question whether or not you need to create a new .torrent, the answer is: yes.

However, depending a bit on the layout of your data, you may be able to do some simple semi-delta-updates.

If the data you distribute is a large collection of individual files, with each build some files may have changed you can simply create a new .torrent file and have all clients download it to the same location as the old one (just like you suggest). The clients would first check the files that already existed on disk, update the ones that had changed and download new files. The main drawback is that removed files would not actually be deleted at the clients.

If you're writing your own client anyway, deleting files on the filesystem that aren't in the .torrent file is a fairly simple step that can be done separately.

This does not work if you distribute an image file, since the bits that stayed the same across the versions may have moved, and thus yielding different piece hashes.

I would not necessarily recommend using super-seeding. Depending on how strict the super seeding implementation you use is, it may actually harm transfer rates. Keep in mind that the purpose of super seeding is to minimize the number of bytes sent from the seed, not to maximize the transfer rate. If all your clients are behaving properly (i.e. using rarest first), the piece distribution shouldn't be a problem anyway.

Also, to create a torrent and to hash-check a 50 GiB torrent puts a lot of load on the drive, you may want to benchmark the bittorrent implementation you use for this, to make sure it's performant enough. At 50 GiB, the difference between different implementations may be significant.

这篇关于使用bittorrent协议夜间分布和CI构建的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！