与大文件Git [英] Git with large files

问题描述

情况

我有两台服务器，即Production and Development。在生产服务器上，有两个应用程序和多个（6）数据库（MySQL），我需要分发给开发人员进行测试。所有源代码都存储在开发服务器上的 GitLab 中，开发人员只能使用此服务器并且无法访问生产服务器。当我们发布一个应用程序时，主要登录到生产环境并从Git中提取新版本。数据库很大（每个数据库超过500M），我需要尽可能地向开发人员分发它们以便测试。

可能的解决方案

• 在将数据库转储到单个文件的备份脚本之后，执行一个将每个数据库推送到其自己的分支的脚本。如果他想更新他的本地副本，开发人员就会选择其中一个分支。

  
  
  

  这一个发现不起作用。




• 生产服务器上的Cron每天保存二进制日志，并将它们推送到该数据库的分支中。所以，在分支中，每天都会有文件发生变化，开发人员会拖拽他没有的文件。当前的SQL转储将以另一种方式发送给开发人员。当存储库的大小变得过大时，我们会将完整转储发送给开发人员并清除存储库中的所有数据，并从头开始。




问题






• 解决方案可能吗？


• 是推/拉仓库，是否上传/下载整个文件，或只是改变它们（即添加新行或编辑当前的行）？


• 可以Git管理如此庞大的文件？没有。
如何设置存储库中保留了多少修订版本？与新解决方案无关。

• 有没有更好的解决方案？我不想强迫开发人员通过FTP或类似方式下载这些大文件。

rsync 可能是有效更新数据库开发者副本的好选择。





它使用增量算法增量更新文件。这样它只会传输已更改或新的文件块。他们当然仍然需要先下载完整的文件，但后来的更新会更快。



基本上，你可以获得与git fetch类似的增量更新，而不需要git克隆将提供的不断扩展的初始副本。损失没有历史，但听起来像你不需要。



rsync是大多数Linux发行版的标准部分，如果你需要它的Windows一个可用的封装端口： http://itefix.no/cwrsync/




要将数据库推送给开发人员，您可以使用类似于以下命令的命令：



  rsync -avz path / to / database（s）HOST：/ folder 
  

或者开发人员可以将数据库需要：

  rsync -avz DATABASE_HOST：/ path / to / database（s）path / where / developer / wants / it 
  




Situation

I have two servers, Production and Development. On Production server, there are two applications and multiple (6) databases (MySQL) which I need to distribute to developers for testing. All source codes are stored in GitLab on Development server and developers are working only with this server and don't have access to production server. When we release an application, master logs into production and pulls new version from Git. The databases are large (over 500M each and counting) and I need to distribute them as easy as possible to developers for testing.

Possible solutions

• After a backup script which dumps databases, each to a single file, execute a script which pushes each database to its own branch. A developer pulls one of these branches if he wants to update his local copy.

  This one was found non working.

• Cron on production server saves binary logs every day and pushes them into the branch of that database. So, in the branch, there are files with daily changes and developer pulls the files he doesn't have. The current SQL dump will be sent to the developer another way. And when the size of the repository becomes too large, we will send full dump to the developers and flush all data in the repository and start from the beginning.

Questions

• Is the solution possible?
• If git is pushing/pulling to/from repository, does it upload/download whole files, or just changes in them (i.e. adds new lines or edits the current ones)?
• Can Git manage so large files? No.
• How to set how many revisions are preserved in a repository? Doesn't matter with the new solution.
• Is there any better solution? I don't want to force the developers to download such large files over FTP or anything similar.

解决方案

rsync could be a good option for efficiently updating the developers copies of the databases.

It uses a delta algorithm to incrementally update the files. That way it only transfers the blocks of the file that have changed or that are new. They will of course still need to download the full file first but later updates would be quicker.

Essentially you get a similar incremental update as a git fetch without the ever expanding initial copy that the git clone would give. The loss is not having the history but is sounds like you don't need that.

rsync is a standard part of most linux distributions if you need it on windows there is a packaged port available: http://itefix.no/cwrsync/

To push the databases to a developer you could use a command similar to:

rsync -avz path/to/database(s) HOST:/folder

Or the developers could pull the database(s) they need with:

rsync -avz DATABASE_HOST:/path/to/database(s) path/where/developer/wants/it

这篇关于与大文件Git的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！