从提交历史记录中删除大于100MB的文件-迁移至Github失败 [英] Remove files larger than 100MB from commit history - migration to Github failing

查看:363
本文介绍了从提交历史记录中删除大于100MB的文件-迁移至Github失败的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!

问题描述

我正在尝试将一个存储库从GitLab迁移到GitHub.存储库大小为685.83MB,它由几个.dat,.csv,.exe,.pkl文件组成,这些文件大于100MB至3383.40 MB.它失败并显示以下错误.

I am trying to Migrate one repository from GitLab to GitHub. The repository size is 685.83MB and it consists of few .dat,.csv,.exe,.pkl files which are more than 100MB to 3383.40 MB. it is failing with below errors.

GitLab To GitHub Migration Steps:-
$ git clone --mirror git@your-gitlab-site.com:test/my-repo.git
$ cd ~/my-repo.git
$ git remote set-url --push origin git@github.com:test/my-repo.git
$ git push

Error
remote: error: GH001: Large files detected. You may want to try Git Large File Storage - https://git-lfs.github.com.
remote: error: File Src/project/label/file1.dat is 476.32 MB; this exceeds GitHub Enterprise's file size limit of 100.00 MB
remote: error: File Src/models/label/file2.dat is 2431.49 MB; this exceeds GitHub Enterprise's file size limit of 100.00 MB
remote: error: File test/test1/label/model/file3.exe is 1031.94 MB; this exceeds GitHub Enterprise's file size limit of 100.00 MB
remote: error: File test/test2/usecase/filemarker/file3.csv is 997.02 MB; this exceeds GitHub Enterprise's file size limit of 100.00 MB
remote: error: File src/msg/sports/model.pkl is 3383.40 MB; this exceeds GitHub Enterprise's file size limit of 100.00 MB
remote: error: File test/movie/maker/marker.dat is 1373.45 MB; this exceeds GitHub Enterprise's file size limit of 100.00 MB
remote: error: File project/make/level/project/realmaker.csv is 1594.83 MB; this exceeds GitHub Enterprise's file size limit of 100.00 MB
remote: error: File src/moderm/network/test.pkl is 111.07 MB; this exceeds GitHub Enterprise's file size limit of 100.00 MB

Git LFS/BFG  Method:
$ git clone --mirror gitlab-heavy-repo 
$ cd gitlab-heavy-repo.git 
$ java -jar bfg-1.12.5.jar --convert-to-git-lfs '*.dat' --no-blob-protection
$ java -jar bfg-1.12.5.jar --convert-to-git-lfs '*.exe' --no-blob-protection
$ java -jar bfg-1.12.5.jar --convert-to-git-lfs '*.csv' --no-blob-protection
$ java -jar bfg-1.12.5.jar --convert-to-git-lfs '*.pkl' --no-blob-protection
$ git reflog expire --expire=now --all && git gc --prune=now
$ git lfs install
$ git remote set-url origin git@github.com:some-org/githubheavy-repo.git
$ git push 

即使经过上述过程,它也会失败,并显示相同的错误.看来Git LFS有2GB的限制.因此,尝试从存储库中完全删除上述较大的文件.请按照以下方法删除.

Even after above process, it fails with same error. it seems Git LFS have 2GB Limitation. So tried to remove the above larger files completely from repository. Followed below method to remove.

1) git clone gitlab-heavy-repo
2) cd gitlab-heavy-repo
3) git filter-branch --force --index-filter "git rm --cached --ignore-unmatch Src/project/label/file1.dat" --prune-empty --tag-name-filter cat -- --all
4) git reflog expire --expire=now --all
5) git gc --prune=now
6) git push origin --force --all
7) git push origin --force --tags
8) rm -rf .git/refs/original/

对以上所有较大的文件重复相同的步骤.但是现在在Gitlab存储库中,存储大小显示-1.9-GB最初只是685.83MB.

Repeated the same steps for all the above larger files. But now in Gitlab repository storage size shows - 1.9-GB initially it was only 685.83MB.

请纠正我.预先感谢.

推荐答案

将100MiB以上的所有文件添加到.gitignore :

find . -size +100M | cat >> .gitignore


如果尚未提交文件,则:

从.gitignore读取文件并将其从回购中删除(而不从磁盘中删除它们):

On Linux:


If you have not committed the files yet:

Read files from .gitignore and remove them from repo (without deleting them from disk):

On Linux:

git ls-files -ci --exclude-standard -z | xargs -0 git rm --cached

On macOS:

alias apply-gitignore="git ls-files -ci --exclude-standard -z | xargs -0 git rm --cached"

On Windows:

for /F "tokens=*" %a in ('git ls-files -ci --exclude-standard') do @git rm --cached "%a"


如果已提交文件:

您需要从提交历史记录中清除它们. 运行以下命令从以前的所有提交中删除文件:


If you have committed the files:

You'll need to clean them from commit history. Run the following command to remove a file from all previous commits:

On Linux and macOS:

git filter-branch --prune-empty -d ~/tmp/scratch \
  --index-filter "git rm --cached -f --ignore-unmatch PATH/TO/FILE" \
  --tag-name-filter cat -- --all

On Windows:

git filter-branch --prune-empty -d /tmp/scratch \
  --index-filter "git rm --cached -f --ignore-unmatch PATH/TO/FILE" \
  --tag-name-filter cat -- --all

(用实际文件的路径替换PATH/TO/FILE)
Greg在此处

(Replace PATH/TO/FILE with path to the actual file)
Greg explains this command better in his answer here

如果您需要对文件夹运行上面的命令而不是文件,请在第二行的git rm之后添加-r开关:

If you need to run the command above for a folder instead of a file, add an -r switch after git rm in the second line:

... \
  --index-filter "git rm -r --cached -f --ignore-unmatch PATH/TO/FOLDER" \
  ...

git rm可以采用多个参数,因此您可以在第二行中添加多个路径:

git rm can take multiple arguments so you can add multiple paths in the second line:

... \
  --index-filter "git rm -r --cached -f --ignore-unmatch FILE1 FILE2 FOLDER1 FOLDER2" \
  ...

这篇关于从提交历史记录中删除大于100MB的文件-迁移至Github失败的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!

查看全文
登录 关闭
扫码关注1秒登录
发送“验证码”获取 | 15天全站免登陆