内存已耗尽：使用diff处理大文件 [英] memory exhausted : for large files using diff

问题描述

我正在尝试使用两个大文件夹（〜7GB）创建补丁。

I am trying to create a patch using two large size folders (~7GB).

这是我的操作方式：

$ diff -Naurbw . ../other-folder > file.patch

但是可能由于文件大小而导致未创建补丁并给出错误：

But maybe due to file sizes, patch is not getting created and giving an error:

diff: memory exhausted

我尝试腾出超过15 GB的空间，但问题仍然存在。

I tried making space more than 15 GB but still the issue persists. Could someone help me out with the flags that I should use?

推荐答案

最近，当我需要区分两个时，我也遇到了这个问题。大文件（每个文件大于5Gb）。

Recently I came across this too when I needed to diff two large files (>5Gb each).

我尝试将'diff'与其他选项一起使用，但即使 --speed-large-files 无效。其他方法包括使用 xdelta 将文件拆分为较小的文件，或按照此建议也没有帮助。我什至可以使用功能非常强大的VM（> 72Gb RAM），但仍然遇到内存耗尽的错误。

I tried to use 'diff' with different options, but even the --speed-large-files had no effect. Other methods like splitting the files into smaller ones, using xdelta or sorting the files as per this suggestion didn't help either. I even got my hands around a very powerful VM (> 72Gb RAM), but still got this memory exhausted error.

我终于通过在 sysctl.conf 中添加以下参数来工作（ sudo vim /etc/sysctl.conf）：

I finally got to work by adding the following parameter to sysctl.conf (sudo vim /etc/sysctl.conf):

vm.overcommit_memory=1

vm.overcommit_memory 具有三个值（0,1,2）并设置内核虚拟内存记帐模式。从 proc（5）手册页：

vm.overcommit_memory has three values (0,1,2) and sets the kernel virtual memory accounting mode. From the proc(5) man page:

0：启发式过量使用（这是默认设置）

1：始终过量使用，从不检查

2：始终检查，永远不要过量使用

0: heuristic overcommit (this is the default)
1: always overcommit, never check
2: always check, never overcommit

要确保确实应用了该参数，您可以运行

To make sure that the parameter is indeed applied you can run

sudo sysctl -p

Don完成操作后，不要忘记更改此参数！

Don't forget to change this parameter back when you finish!

这篇关于内存已耗尽：使用diff处理大文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！