对于大于100MB的文件，git lfs无法正常工作 [英] git lfs not working properly for files larger than 100MB

问题描述

我被git建议使用git lfs处理大文件。在我用git lfs跟踪它们并检查它们是否添加到.gitattribute后，我仍然得到文件大于100MB的相同确切文件的错误。这里有什么建议以及我如何解决这个问题？我需要将这些大文件作为项目的一部分上传到github。

  jalal @ klein：〜/ computer_vision / py-faster-rcnn $ git push -u'https://github.com'的masterUsername：monajalal 
'https：//[e-mail]'的密码：
计数对象：65949 ，完成。 
使用多达40个线程的增量压缩。 
压缩对象：100％（64616/64616），完成。 
写作对象：100％（65949/65949），1.53 GiB | 11.18 MiB / s，完成。 
合计65949（增量29371），重用2（增量0）
远程：解决增量：100％（29371/29371），完成。 
 remote：错误：GH001：检测到大文件。你可能想尝试Git Large File Storage  -  https://git-lfs.github.com。 
 remote：error：跟踪：b3969bd24e3d0b92cab7299bbf735509 
 remote：错误：有关更多信息，请参阅http://git.io/iEPt8g。 
 remote：error：File caffe-fast-rcnn / boost_1_61_0 / bin.v2 / libs / math / build / gcc-5.2.1 / release / link-static / src / tr1 / pch.hpp.gch是127.21 MB;这超过了GitHub的文件大小限制为100.00 MB 
 remote：error：文件数据/ faster_rcnn_models / ZF_faster_rcnn_final.caffemodel为226.19 MB;这超出了GitHub的文件大小限制为100.00 MB 
 remote：错误：文件数据/ faster_rcnn_models.tgz是694.81 MB;这超过了GitHub的文件大小限制为100.00 MB 
 remote：error：文件数据/ faster_rcnn_models / VGG16_faster_rcnn_final.caffemodel为522.92 MB;这超过了GitHub的文件大小限制为100.00MB 
 remote：error：file caffe-fast-rcnn / boost_1_61_0 / bin.v2 / libs / math / build / gcc-5.2.1 / release / src / tr1 / pch。 hpp.gch是127.21 MB;这超过了GitHub的文件大小限制为100.00 MB 
到https://github.com/monajalal/faster_RCNN.git 
！ [remote rejected] master  - > master（pre-receive hook refused）
错误：未能将一些文件推送到'https://github.com/monajalal/faster_RCNN.git'
 
 
 
 jalal @ klein：〜/ computer_vision / py-faster-rcnn $ ls 
 caffe-fast-rcnn数据实验lib LICENSE模型README.md测试工具
 jalal @ klein：〜/ computer_vision / py-faster -rcnn $ git lfs track* .gch
跟踪* .gch 
 jalal @ klein：〜/ computer_vision / py-faster-rcnn $ git lfs track* .tgz
跟踪* .tgz 
 jalal @ klein：〜/ computer_vision / py-faster-rcnn $ git lfs track* .caffemodel
跟踪* .caffemodel 
 jalal @ klein：〜/ computer_vision / py -faster-rcnn $ git add -A 
 jalal @ klein：〜/ computer_vision / py-faster-rcnn $ git commit -m使用git lfs跟踪的大于100MB的文件
 [master 
 b60c05f]使用git lfs跟踪的大于100MB的文件
 6个文件已更改，3个插入（+）
创建模式100644 .gitattributes 
 rewrite caffe-fast-rcnn / boost_1_61_0 / bin.v2 /库/数学/建设/ GCC-5.2.1 /发行/链接-STAT ic / src / tr1 / pch.hpp.gch（99％）
 rewrite caffe-fast-rcnn / boost_1_61_0 / bin.v2 / libs / math / build / gcc-5.2.1 / release / src / tr1 / （99％）
重写data / faster_rcnn_models.tgz（99％）
重写数据/ faster_rcnn_models / VGG16_faster_rcnn_final.caffemodel（99％）
重写数据/ faster_rcnn_models / ZF_faster_rcnn_final。 caffemodel（99％）
 jalal @ klein：〜/ computer_vision / py-faster-rcnn $ 
 jalal @ klein：〜/ computer_vision / py-faster-rcnn $ ls -a 
。 .. caffe-fast-rcnn数据实验.git .gitattributes lib LICENSE模型README.md测试工具
 
 $ b jalal @ klein：〜/ computer_vision / py-faster-rcnn $ git push  - u原产地master 
'https://github.com'的用户名：monajalal 
'https：//[e-mail]'的密码：
 https：// github的用户名.com'：monajalal 
密码为'https：//[e-mail]'：
 Git LFS：（5个共5个文件）1.66 GB / 1.66 GB 
计数对象：65972，完成。 
使用多达40个线程的增量压缩。 
压缩对象：100％（64633/64633），完成。 
写作对象：100％（65972/65972），1.53 GiB | 11.18 MiB / s，完成。 
共计65972（delta 29378），重用2（delta 0）
 remote：解决三角洲：100％（29378/29378），已完成。 
 remote：错误：GH001：检测到大文件。你可能想尝试Git Large File Storage  -  https://git-lfs.github.com。 
 remote：错误：跟踪：4acfe1a1fdf5874edf9141fa8327428d 
 remote：错误：有关更多信息，请参阅http://git.io/iEPt8g。 
 remote：error：File caffe-fast-rcnn / boost_1_61_0 / bin.v2 / libs / math / build / gcc-5.2.1 / release / link-static / src / tr1 / pch.hpp.gch是127.21 MB;这超过了GitHub的文件大小限制为100.00 MB 
 remote：error：文件数据/ faster_rcnn_models / ZF_faster_rcnn_final.caffemodel为226.19 MB;这超出了GitHub的文件大小限制为100.00 MB 
 remote：错误：文件数据/ faster_rcnn_models.tgz是694.81 MB;这超过了GitHub的文件大小限制为100.00 MB 
 remote：error：文件数据/ faster_rcnn_models / VGG16_faster_rcnn_final.caffemodel为522.92 MB;这超过了GitHub的文件大小限制为100.00MB 
 remote：error：file caffe-fast-rcnn / boost_1_61_0 / bin.v2 / libs / math / build / gcc-5.2.1 / release / src / tr1 / pch。 hpp.gch是127.21 MB;这超过了GitHub的文件大小限制为100.00 MB 
到https://github.com/monajalal/faster_RCNN.git 
！ [remote rejected] master  - > master（pre-receive hook refused）
错误：未能将一些文件推送到'https://github.com/monajalal/faster_RCNN.git'
 jalal @ klein：〜/ computer_vision / py-faster -rcnn $ 
  

里面 .gitattribute file I有：

  * .gch filter = lfs diff = lfs merge = lfs -text 
 * .tgz filter = lfs diff = lfs merge = lfs -text 
 * .caffemodel filter = lfs diff = lfs merge = lfs -text 
  

git bfg 是一个工具（用java编写），可以轻松从历史记录中删除较大的文件。

只使用内置工具，您可以使用git 过滤器-branch 与 - tree-filter 删除文件（示例应该在文档中）。

注意，像BFG和 git filter-branch 这样的工具，重写历史。如果您重写您推送的提交，这可能会影响构建这些提交的其他人。小心使用。

I was suggested by git to use git lfs for large files. After I tracked them with git lfs and checked if they are added to .gitattribute I still get the error that files are larger than 100MB for the same exact files. What are the suggestions here and how I can solve this problem? I would need to upload these large files as part of the project to github as well.

    jalal@klein:~/computer_vision/py-faster-rcnn$ git push -u origin masterUsername for 'https://github.com': monajalal
    Password for 'https://[e-mail]': 
    Counting objects: 65949, done.
    Delta compression using up to 40 threads.
    Compressing objects: 100% (64616/64616), done.
    Writing objects: 100% (65949/65949), 1.53 GiB | 11.18 MiB/s, done.
    Total 65949 (delta 29371), reused 2 (delta 0)
    remote: Resolving deltas: 100% (29371/29371), done.
    remote: error: GH001: Large files detected. You may want to try Git Large File Storage - https://git-lfs.github.com.
    remote: error: Trace: b3969bd24e3d0b92cab7299bbf735509
    remote: error: See http://git.io/iEPt8g for more information.
    remote: error: File caffe-fast-rcnn/boost_1_61_0/bin.v2/libs/math/build/gcc-5.2.1/release/link-static/src/tr1/pch.hpp.gch is 127.21 MB; this exceeds GitHub's file size limit of 100.00 MB
    remote: error: File data/faster_rcnn_models/ZF_faster_rcnn_final.caffemodel is 226.19 MB; this exceeds GitHub's file size limit of 100.00 MB
    remote: error: File data/faster_rcnn_models.tgz is 694.81 MB; this exceeds GitHub's file size limit of 100.00 MB
    remote: error: File data/faster_rcnn_models/VGG16_faster_rcnn_final.caffemodel is 522.92 MB; this exceeds GitHub's file size limit of 100.00 MB
    remote: error: File caffe-fast-rcnn/boost_1_61_0/bin.v2/libs/math/build/gcc-5.2.1/release/src/tr1/pch.hpp.gch is 127.21 MB; this exceeds GitHub's file size limit of 100.00 MB
    To https://github.com/monajalal/faster_RCNN.git
     ! [remote rejected] master -> master (pre-receive hook declined)
    error: failed to push some refs to 'https://github.com/monajalal/faster_RCNN.git'



jalal@klein:~/computer_vision/py-faster-rcnn$ ls
caffe-fast-rcnn  data  experiments  lib  LICENSE  models  README.md  test  tools
jalal@klein:~/computer_vision/py-faster-rcnn$ git lfs track "*.gch"
Tracking *.gch
jalal@klein:~/computer_vision/py-faster-rcnn$ git lfs track "*.tgz"
Tracking *.tgz
jalal@klein:~/computer_vision/py-faster-rcnn$ git lfs track "*.caffemodel"
Tracking *.caffemodel
jalal@klein:~/computer_vision/py-faster-rcnn$ git add -A
jalal@klein:~/computer_vision/py-faster-rcnn$ git commit -m "files larger than 100MB tracked with git lfs"
[master 
b60c05f] files larger than 100MB tracked with git lfs
 6 files changed, 3 insertions(+)
 create mode 100644 .gitattributes
 rewrite caffe-fast-rcnn/boost_1_61_0/bin.v2/libs/math/build/gcc-5.2.1/release/link-static/src/tr1/pch.hpp.gch (99%)
 rewrite caffe-fast-rcnn/boost_1_61_0/bin.v2/libs/math/build/gcc-5.2.1/release/src/tr1/pch.hpp.gch (99%)
 rewrite data/faster_rcnn_models.tgz (99%)
 rewrite data/faster_rcnn_models/VGG16_faster_rcnn_final.caffemodel (99%)
 rewrite data/faster_rcnn_models/ZF_faster_rcnn_final.caffemodel (99%)
jalal@klein:~/computer_vision/py-faster-rcnn$ 
jalal@klein:~/computer_vision/py-faster-rcnn$ ls -a
.  ..  caffe-fast-rcnn  data  experiments  .git  .gitattributes  lib  LICENSE  models  README.md  test  tools


jalal@klein:~/computer_vision/py-faster-rcnn$ git push -u origin master
Username for 'https://github.com': monajalal
Password for 'https://[e-mail]': 
Username for 'https://github.com': monajalal
Password for 'https://[e-mail]': 
Git LFS: (5 of 5 files) 1.66 GB / 1.66 GB                                                                                                             
Counting objects: 65972, done.
Delta compression using up to 40 threads.
Compressing objects: 100% (64633/64633), done.
Writing objects: 100% (65972/65972), 1.53 GiB | 11.18 MiB/s, done.
Total 65972 (delta 29378), reused 2 (delta 0)
remote: Resolving deltas: 100% (29378/29378), done.
remote: error: GH001: Large files detected. You may want to try Git Large File Storage - https://git-lfs.github.com.
remote: error: Trace: 4acfe1a1fdf5874edf9141fa8327428d
remote: error: See http://git.io/iEPt8g for more information.
remote: error: File caffe-fast-rcnn/boost_1_61_0/bin.v2/libs/math/build/gcc-5.2.1/release/link-static/src/tr1/pch.hpp.gch is 127.21 MB; this exceeds GitHub's file size limit of 100.00 MB
remote: error: File data/faster_rcnn_models/ZF_faster_rcnn_final.caffemodel is 226.19 MB; this exceeds GitHub's file size limit of 100.00 MB
remote: error: File data/faster_rcnn_models.tgz is 694.81 MB; this exceeds GitHub's file size limit of 100.00 MB
remote: error: File data/faster_rcnn_models/VGG16_faster_rcnn_final.caffemodel is 522.92 MB; this exceeds GitHub's file size limit of 100.00 MB
remote: error: File caffe-fast-rcnn/boost_1_61_0/bin.v2/libs/math/build/gcc-5.2.1/release/src/tr1/pch.hpp.gch is 127.21 MB; this exceeds GitHub's file size limit of 100.00 MB
To https://github.com/monajalal/faster_RCNN.git
 ! [remote rejected] master -> master (pre-receive hook declined)
error: failed to push some refs to 'https://github.com/monajalal/faster_RCNN.git'
jalal@klein:~/computer_vision/py-faster-rcnn$

Inside .gitattribute file I have:

    *.gch filter=lfs diff=lfs merge=lfs -text
    *.tgz filter=lfs diff=lfs merge=lfs -text
    *.caffemodel filter=lfs diff=lfs merge=lfs -text

解决方案

Your history still contains the large files, which you need to rewrite to remove.

git bfg is a tool (written in java) to easily remove larger files from history.

Using only built-in tools, you can use git filter-branch with --tree-filter to remove the file (examples should be in the doc).

Warning:

Note that tools like BFG and git filter-branch rewrite history. If you rewrite commits that you have pushed, this can affect others who build on those commits. Use with care.

这篇关于对于大于100MB的文件，git lfs无法正常工作的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！