mmap缓存 [英] mmap caching
问题描述
我一直试图追踪内存泄漏（我最初将b
$ b错误归因于numpy）并且结果是由于内存泄漏造成的内存泄漏映射文件。似乎mmap缓存没有限制它读取的块，因为在内存使用量因MemoryError死亡之前，内存使用量增加到几百MB，相当于Windows任务管理器。我是b / b
这些块在其他任何地方都没有被引用;实际上如果我将bmap对象更改为普通文件，那么内存使用量仍然是
常量。 mmap的文档没有提到
这个。可以在用户级别修改缓存策略吗？
George
George Sakkis< ge *********** @ gmail.comwrote：
我一直试图追踪内存泄漏（其中我最初
错误归因于numpy）并且结果是由一个
内存映射文件引起的。似乎mmap缓存没有限制它读取的块，因为在内存使用量因MemoryError死亡之前，内存使用量增加到几百MB，相当于Windows任务管理器。我是b / b
这些块在其他任何地方都没有被引用;实际上如果我将bmap对象更改为普通文件，那么内存使用量仍然是
常量。 mmap的文档没有提到
这个。可以在用户级别修改缓存策略吗？
我不熟悉windows上的mmap（），但假设它的工作方式与unix相同......
... br />
mmap（）的目的是将文件映射到内存中。操作系统完全可以将页面带入内存供您读取/写入，并且完全可以通过操作系统将b $ b再次删除。
你所期望的是文件被需求分页到内存中，因为你可以访问它的位数。这些页面将保留在内存中，直到操作系统
感觉到一些内存压力，如果页面将被写出来，如果脏了
然后丢弃。
操作系统将尽可能长时间保持页面，以防万一你再次需要它们。丢失的页面应该是最近使用的页面。
我不会期待MemoryError虽然...
你写完后写了mmap.flush（）吗？
-
Nick Craig-Wood< ni ** @ craig -wood.com-- http://www.craig-wood.com/nick
Nick Craig-Wood写道：
George Sakkis< ge ***** ******@gmail.com写的：
我一直在试图追踪内存泄漏（我最初的确认为
错误地归因于numpy）并且结果是由
内存映射文件引起的。似乎mmap缓存没有限制它读取的块，因为在内存使用量因MemoryError死亡之前，内存使用量增加到几百MB，相当于Windows任务管理器。我是b / b
这些块在其他任何地方都没有被引用;实际上如果我将bmap对象更改为普通文件，那么内存使用量仍然是
常量。 mmap的文档没有提到
这个。可以在用户级别修改缓存策略吗？
我不熟悉windows上的mmap（），但假设它的工作方式与unix相同......
...... br />
mmap（）的目的是将文件映射到内存中。操作系统完全可以将页面带入内存供您读取/写入，并且完全可以通过操作系统将b $ b再次删除。
你所期望的是文件被需求分页到内存中，因为你可以访问它的位数。这些页面将保留在内存中，直到操作系统
感觉到一些内存压力，如果页面将被写出来，如果脏了
然后丢弃。
操作系统将尽可能长时间保持页面，以防万一你再次需要它们。丢失的页面应该是最近使用的页面。
我不会期待MemoryError虽然...
写完后你做过mmap.flush（）吗？
文件写入一次然后以只读方式打开，没有
刷新。因此，如果缓存完全取决于操作系统，我认为我的
选项要么（1）修改我的算法，以便它们在
固定大小的批次中工作任意长序列或（2）
实现我自己的内存映射方案以适合我的算法。我想
（1）总体来说会不那么麻烦，或者有没有办法给操作系统提供一个关于它可以使用多大缓存的提示？b br />
George
George Sakkis schrieb：
我一直试图追踪内存泄漏（我最初将错误地归因于numpy），结果是由一个内存映射文件引起的内存泄漏。似乎mmap缓存没有限制它读取的块，因为在内存使用量因MemoryError死亡之前，内存使用量增加到几百MB，相当于Windows任务管理器。
你必须误解你所看到的。这是运行
系统决定内存映射文件的哪一部分保存在内存中，这当然不是没有限制的。
请注意，有几个值可以称为内存
使用 （例如承诺地址空间的大小，工作
设置大小等）;你没有提到这些价值中的哪些增长了几百元。
问候，
马丁
I''ve been trying to track down a memory leak (which I initially
attributed erroneously to numpy) and it turns out to be caused by a
memory mapped file. It seems that mmap caches without limit the chunks
it reads, as the memory usage grows to several hundreds MBs according
to the Windows task manager before it dies with a MemoryError. I''m
positive that these chunks are not referenced anywhere else; in fact if
I change the mmap object to a normal file, memory usage remains
constant. The documentation of mmap doesn''t mention anything about
this. Can the caching strategy be modified at the user level ?
George
George Sakkis <ge***********@gmail.comwrote:I''ve been trying to track down a memory leak (which I initially
attributed erroneously to numpy) and it turns out to be caused by a
memory mapped file. It seems that mmap caches without limit the chunks
it reads, as the memory usage grows to several hundreds MBs according
to the Windows task manager before it dies with a MemoryError. I''m
positive that these chunks are not referenced anywhere else; in fact if
I change the mmap object to a normal file, memory usage remains
constant. The documentation of mmap doesn''t mention anything about
this. Can the caching strategy be modified at the user level ?I''m not familiar with mmap() on windows, but assuming it works the
same way as unix...
The point of mmap() is to map files into memory. It is completely up
to the OS to bring pages into memory for you to read / write to, and
completely up to the OS to get rid of them again.
What you would expect is that the file is demand paged into memory as
you access bits of it. These pages will remain in memory until the OS
feels some memory pressure when the pages will be written out if dirty
and then dropped.
The OS will try to keep hold of pages as long as possible just in case
you need them again. The pages dropped should be the least recently
used pages.
I wouldn''t have expected a MemoryError though...
Did you do mmap.flush() after writing?
--
Nick Craig-Wood <ni**@craig-wood.com-- http://www.craig-wood.com/nick
Nick Craig-Wood wrote:
George Sakkis <ge***********@gmail.comwrote:I''ve been trying to track down a memory leak (which I initially
attributed erroneously to numpy) and it turns out to be caused by a
memory mapped file. It seems that mmap caches without limit the chunks
it reads, as the memory usage grows to several hundreds MBs according
to the Windows task manager before it dies with a MemoryError. I''m
positive that these chunks are not referenced anywhere else; in fact if
I change the mmap object to a normal file, memory usage remains
constant. The documentation of mmap doesn''t mention anything about
this. Can the caching strategy be modified at the user level ?
I''m not familiar with mmap() on windows, but assuming it works the
same way as unix...
The point of mmap() is to map files into memory. It is completely up
to the OS to bring pages into memory for you to read / write to, and
completely up to the OS to get rid of them again.
What you would expect is that the file is demand paged into memory as
you access bits of it. These pages will remain in memory until the OS
feels some memory pressure when the pages will be written out if dirty
and then dropped.
The OS will try to keep hold of pages as long as possible just in case
you need them again. The pages dropped should be the least recently
used pages.
I wouldn''t have expected a MemoryError though...
Did you do mmap.flush() after writing?The file is written once and then opened as read-only, there''s no
flushing. So if caching is completely up to the OS, I take it that my
options are either (1) modify my algorithms so that they work in
fixed-size batches instead of arbitrarily long sequences or (2)
implement my own memory-mapping scheme to fit my algorithms. I guess
(1) would be the less trouble overall, or is there a way to give a hint
to the OS on how large cache can it use ?
George
George Sakkis schrieb:I''ve been trying to track down a memory leak (which I initially
attributed erroneously to numpy) and it turns out to be caused by a
memory mapped file. It seems that mmap caches without limit the chunks
it reads, as the memory usage grows to several hundreds MBs according
to the Windows task manager before it dies with a MemoryError.You must be misinterpreting what you are seeing. It''s the operating
system that decides what part of a memory-mapped file are held in
memory, and that is certainly not without limits.
Notice that there are several values that can be called "memory
usage" (such as the size of the committed address space, the working
set size, etc); you don''t mention which of these values grows several
hundreds MB.
Regards,
Martin
这篇关于mmap缓存的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
