如何限制复制所使用的缓存，以便仍然有其他缓存可用的内存? [英] How can I limit the cache used by copying so there is still memory available for other caches?

问题描述

基本情况:

我正在 openSUSE 中复制一些NTFS磁盘.每个是2 TB.当我这样做时，系统运行缓慢.

I am copying some NTFS disks in openSUSE. Each one is 2 TB. When I do this, the system runs slow.

我的猜测:

我认为这可能是由于缓存所致.Linux决定放弃有用的缓存(例如， KDE4 膨胀，虚拟机磁盘， LibreOffice 二进制文件，Thunderbird二进制文件等)，而是在其中填充所有可用内存(总计24 GB)从复制磁盘中读取，复制磁盘将只读取一次，然后写入，并且不再使用.因此，每当我使用这些应用程序(或KDE 4)时，都需要再次读取磁盘，并且再次读取磁盘上的膨胀文件会使事情冻结/打ic.

I believe it is likely due to caching. Linux decides to discard useful caches (for example, KDE 4 bloat, virtual machine disks, LibreOffice binaries, Thunderbird binaries, etc.) and instead fill all available memory (24 GB total) with stuff from the copying disks, which will be read only once, then written and never used again. So then any time I use these applications (or KDE 4), the disk needs to be read again, and reading the bloat off the disk again makes things freeze/hiccup.

由于缓存消失了，并且这些ated肿的应用程序需要大量缓存，这使系统非常慢.

Due to the cache being gone and the fact that these bloated applications need lots of cache, this makes the system horribly slow.

由于它是USB，所以磁盘和磁盘控制器不是瓶颈，因此请使用 ionice 不能使其更快.

Since it is USB, the disk and disk controller are not the bottleneck, so using ionice does not make it faster.

我相信这是缓存，而不仅仅是主板运行太慢，因为如果我停止复制所有内容，它仍然会运行一段时间，直到重新缓存所有内容为止.

I believe it is the cache rather than just the motherboard going too slow, because if I stop everything copying, it still runs choppy for a while until it recaches everything.

如果我重新开始复制​​，则需要一分钟才能再次断断续续.而且，我可以将其限制在40 MB/s左右，并且再次运行得更快(不是因为它缓存了正确的东西，而是因为主板总线为系统磁盘提供了很多额外的带宽).我完全可以接受由于主板的I/O功能被完全消耗(已使用100％，这意味着0％的功耗使我感到高兴)而导致的性能损失，但是我不能接受这种缓存机制在此方面的表现如此糟糕特定的用例.

And if I restart the copying, it takes a minute before it is choppy again. But also, I can limit it to around 40 MB/s, and it runs faster again (not because it has the right things cached, but because the motherboard busses have lots of extra bandwidth for the system disks). I can fully accept a performance loss from my motherboard's I/O capability being completely consumed (which is 100% used, meaning 0% wasted power which makes me happy), but I can't accept that this caching mechanism performs so terribly in this specific use case.

# free
             total       used       free     shared    buffers     cached
Mem:      24731556   24531876     199680          0    8834056   12998916
-/+ buffers/cache:    2698904   22032652
Swap:      4194300      24764    4169536

我也在Ubuntu上尝试了相同的方法，这导致整个系统挂起.;)

I also tried the same thing on Ubuntu, which causes a total system hang instead. ;)

为了澄清，我不是在问如何为系统"留出空闲的存储空间，而是为高速缓存"留出空闲的存储空间.我知道缓存会在需要时自动分配给系统，但是我的问题是它不保留用于缓存特定的东西.

And to clarify, I am not asking how to leave memory free for the "system", but for "cache". I know that cache memory is automatically given back to the system when needed, but my problem is that it is not reserved for caching of specific things.

是否有某种方法可以告诉这些复制操作以限制内存使用，从而使某些重要的东西保持高速缓存，因此，任何速度下降都是由于正常的磁盘使用而导致的，而不是重新读取相同的常用文件?例如，是否设置了允许每个进程/用户/文件系统用作缓存/缓冲区的最大内存?

Is there some way to tell these copy operations to limit memory usage so some important things remain cached, and therefore any slowdowns are a result of normal disk usage and not rereading the same commonly used files? For example, is there a setting of max memory per process/user/file system allowed to be used as cache/buffers?

推荐答案

nocache 命令是这个问题的一般答案！它也在Debian和 Ubuntu 13.10 (Saucy Salamander)中

The nocache command is the general answer to this problem! It is also in Debian and Ubuntu 13.10 (Saucy Salamander).

谢谢彼得，提醒我们注意--drop-cache"rsync中的选项.但这在上游被拒绝(错误9560 –删除缓存选项)，为此，我们倾向于使用更通用的解决方案:新的"nocache"基于rsync的命令与fadvise配合使用.

Thanks, Peter, for alerting us to the --drop-cache" option in rsync. But that was rejected upstream (Bug 9560 – drop-cache option), in favor of a more general solution for this: the new "nocache" command based on the rsync work with fadvise.

您只需在前面添加"nocache"即可到您想要的任何命令.它还具有用于描述和修改文件的缓存状态的实用程序.例如，这是带有和不带有nocache的效果:

You just prepend "nocache" to any command you want. It also has nice utilities for describing and modifying the cache status of files. For example, here are the effects with and without nocache:

$ ./cachestats ~/file.mp3
pages in cache: 154/1945 (7.9%)  [filesize=7776.2K, pagesize=4K]
$ ./nocache cp ~/file.mp3 /tmp
$ ./cachestats ~/file.mp3
pages in cache: 154/1945 (7.9%)  [filesize=7776.2K, pagesize=4K]\
$ cp ~/file.mp3 /tmp
$ ./cachestats ~/file.mp3
pages in cache: 1945/1945 (100.0%)  [filesize=7776.2K, pagesize=4K]

因此，希望它对其他备份程序(rsnapshot，duplicity，rdiff-backup，amanda，s3sync，s3ql，tar等)和您不希望破坏高速缓存的其他命令起作用.

So hopefully that will work for other backup programs (rsnapshot, duplicity, rdiff-backup, amanda, s3sync, s3ql, tar, etc.) and other commands that you don't want trashing your cache.

这篇关于如何限制复制所使用的缓存，以便仍然有其他缓存可用的内存?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！