如何限制复制使用的缓存，以便其他缓存仍有内存可用？ [英] How can I limit the cache used by copying so there is still memory available for other cache?

问题描述

基本情况：

我在openSuSE中复制一些NTFS磁盘。每个是2TB。当我这样做时，系统运行缓慢。

我的猜测是：

我相信这可能是由于缓存。 Linux决定放弃有用的缓存（例如，kde4膨胀，虚拟机磁盘，LibreOffice二进制文件，Thunderbird二进制文件等），而是用复制磁盘中的东西填充所有可用的内存（总共24 GB），这些内容只能读取一次，然后写，永远不会再使用。因此，任何时候我使用这些应用程序（或KDE4），磁盘需要再次读取，再次读取磁盘膨胀使事情冻结/打嗝。

由于缓存不存在，而且这些臃肿的应用程序需要大量的缓存，这使得系统变得非常慢。

因为它是USB，磁盘和磁盘控制器不是瓶颈，所以使用ionice不会让它更快。

我相信这是缓存，而不是只是主板太慢，因为如果我停止一切复制，它仍然跑了一段时间，直到它recache一切。如果我重新开始复制​​，它需要一分钟才能再次波动。而且，我可以将其限制在40 MB / s左右，而且运行速度更快（不是因为它有正确的缓存，而是因为主板总线为系统磁盘提供了额外的带宽）。我可以完全接受主板IO功能被完全消耗（这是100％的使用，这意味着0％的浪费的功率，这使我感到高兴）的性能损失，但我不能接受，这种缓存机制在这个具体的使用

 ＃free 
使用的空闲共享缓冲区缓存
 Mem：24731556 24531876 199680 0 8834056 12998916 
  -  / + buffers / cache：2698904 22032652 
掉期：4194300 24764 4169536 
  

我也在Ubuntu上试过同样的东西，这会导致整个系统挂起。 ;）

为了澄清，我不是问如何为系统留下内存空间，而是为了缓存。我知道在需要的时候缓冲存储器会自动返回给系统，但是我的问题是它并不保留用于缓存特定的事情。

问题：有什么方法可以告诉这些复制操作限制内存的使用，所以一些重要的东西仍然被缓存，因此任何减速都是正常磁盘使用的结果而不重读相同的常用文件？例如，是否允许每个进程/用户/文件系统允许用作缓存/缓冲区的最大内存设置？

解决方案

Kristof Provost非常接近，但在我的情况下，我不想使用dd或编写自己的软件，所以解决方案是使用rsync中的--drop-cache选项。

自从创建这个问题以来，我使用了很多次，似乎完全解决了这个问题。当我使用rsync从不支持--drop-cache的FreeBSD机器上复制时，有一个例外。所以我写了一个包装来替换/ usr / local / bin / rsync命令，并删除该选项，现在它也可以从那里复制。

它仍然使用巨大的内存缓冲区，似乎几乎没有缓存，但工作顺利反正。

  $ free 
总使用空闲共享缓冲区缓存
 Mem：24731544 24531576 199968 0 15349680 850624 
  -  / + buffers / cache：8331272 16400272 
 Swap：4194300 602648 3591652 
  

Basic situation:

I am copying some NTFS disks in openSuSE. Each one is 2TB. When I do this, the system runs slow.

My guesses:

I believe it is likely due to caching. Linux decides to discard useful cache (eg. kde4 bloat, virtual machine disks, LibreOffice binaries, Thunderbird binaries, etc.) and instead fill all available memory (24 GB total) with stuff from the copying disks, which will be read only once, then written and never used again. So then any time I use these apps (or kde4), the disk needs to be read again, and reading the bloat off the disk again makes things freeze/hiccup.

Due to the cache being gone and the fact that these bloated applications need lots of cache, this makes the system horribly slow.

Since it is USB,the disk and disk controller are not the bottleneck, so using ionice does not make it faster.

I believe it is the cache rather than just the motherboard going too slow, because if I stop everything copying, it still runs choppy for a while until it recaches everything. And if I restart the copying, it takes a minute before it is choppy again. But also, I can limit it to around 40 MB/s, and it runs faster again (not because it has the right things cached, but because the motherboard busses have lots of extra bandwidth for the system disks). I can fully accept a performance loss from my motherboard's IO capability being completely consumed (which is 100% used, meaning 0% wasted power which makes me happy), but I can't accept that this caching mechanism performs so terribly in this specific use case.

# free
             total       used       free     shared    buffers     cached
Mem:      24731556   24531876     199680          0    8834056   12998916
-/+ buffers/cache:    2698904   22032652
Swap:      4194300      24764    4169536

I also tried the same thing on Ubuntu, which causes a total system hang instead. ;)

And to clarify, I am not asking how to leave memory free for the "system", but for "cache". I know that cache memory is automatically given back to the system when needed, but my problem is that it is not reserved for caching of specific things.

Question:

Is there some way to tell these copy operations to limit memory usage so some important things remain cached, and therefore any slowdowns are a result of normal disk usage and not rereading the same commonly used files? For example, is there a setting of max memory per process/user/file system allowed to be used as cache/buffers?

解决方案

Kristof Provost was very close, but in my situation, I didn't want to use dd or write my own software, so the solution was to use the "--drop-cache" option in rsync.

I have used this many times since creating this question, and it seems to fix the problem completely. One exception was when I am using rsync to copy from a FreeBSD machine, which doesn't support "--drop-cache". So I wrote a wrapper to replace the /usr/local/bin/rsync command, and remove that option, and now it works copying from there too.

It still uses huge amount of memory for buffers, and seems to keep almost no cache, but works smoothly anyway.

$ free
             total       used       free     shared    buffers     cached
Mem:      24731544   24531576     199968          0   15349680     850624
-/+ buffers/cache:    8331272   16400272
Swap:      4194300     602648    3591652

这篇关于如何限制复制使用的缓存，以便其他缓存仍有内存可用？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！