文件空洞和共享内存在Linux呢？ [英] File Holes and Shared Memory in Linux?

问题描述

由于我阅读Linux编程接口书（非常好读）我发现在Linux中的文件孔。因此，使用Unix文件格式，如的Ext4可以打开一个新文件的写入1000字节谋求1000.000.000写入另一个1000个字节的位置，并根据该文件格式的块大小与文件消耗为块大小只2048字节结束1024或512。

所以基本上该文件作为其中使用的实际驱动器空间只有两个街区了1GB + 1000字节长的文件创建的。

我可以删除一个文件的中间强制系统释放驱动器上的那些块？

有哪里分配（共享存储器）具有或不具有文件备用它在那里它有也被刚填充的存储器页面写入孔的等效？

将是很好的分配1GB的共享内存，但从来没有充分利用它，直到有必要，只是为了避免重映射如果共享内存块要长。

解决方案

  

我可以删除一个文件的中间强制系统释放驱动器上的那些块？

您可能希望在Linux特有的 fallocate（2） ;当心，它可能无法在某些文件系统（例如NFS，VFAT，...）工作，因为有些文件系统不要有破洞。另请参见 lseek的（2） SEEK_HOLE posix_fadvise（2） ，的madvise（2），的 memfd_create（2），等等...

块设备（如磁盘分区，或USB密钥，或SSD）没有任何孔（但你可以 MMAP 放进去）。孔是一个文件系统软件神器。

  

将是很好的分配1GB的共享内存，但从来没有使用它

这是矛盾的。如果内存的共享的它的使用的（用你的的东西-generally另一个流程 - 共享的内存）。阅读 shm_overview（7）如果你真的想要的共享内存（和读取的仔细的的的mmap（2））。了解更多关于虚拟内存，的地址空间，分页，<一个HREF =htt​​ps://en.wikipedia.org/wiki/Memory_management_unit相对=nofollow> MMU S，的页面错误，操作系统中的内核，的 MMAP ，按需分页，的写入时复制，的内存地图， ELF ，的稀疏文件 ...尝试也在执行cat / proc / $$ /图在终端命令，了解输出（请参见 PROC（5） ...）。

也许你想pre-分配一些地址空间范围，后来真的分配虚拟内存。这可能是使用Linux版本的mmap（2）的。

要pre-分配技嘉内存范围，你会首先调用 MMAP 与 MAP_NORESERVE

 为size_t onegiga = 1L＆LT;＆LT; 30;
无效* startad = MMAP（NULL，onegiga，PROT_NONE，
                MAP_ANONYMOUS | MAP_NORESERVE | MAP_SHARED，
                -1，0）;
如果（startad == MAP_FAILED）{PERROR（MMAP MAP_NORESERVE）;出口（EXIT_FAILURE）; }
无效* ENDAD =（字符*）startad + onegiga;
 

的 MAP_NORESERVE 不消耗大量的资源（即不吃饭交换空间，这是不保留，旗故名）。这是pre-分配的地址空间，在这个意义上，进一步 MMAP 通话（不含 MAP_FIXED ）荣获 ŧ给返回范围内的地址（除非你则munmap 它的一些）。

之后，你可以分配的一些子段，在页面大小（一般为4KB）的整数倍，使用 MAP_FIXED 的previous段内，如

 为size_t segoff = 1024 * 1024; //或别的东西，这样....
断言（segoff＆GT; = 0＆放大器;＆安培; segoff＆LT; onegiga＆放大器;＆安培; segoff％的sysconf（_SC_PAGESIZE）== 0）;
为size_t SEGSIZE = 65536; //或别的东西，这样....
断言（SEGSIZE大于0和放大器;＆安培; SEGSIZE％的sysconf（_SC_PAGESIZE）== 0
        ＆功放;＆安培; startad + segoffset + SEGSIZE＆LT; ENDAD）;
无效* segmentad = MMAP（startad + segoffset，SEGSIZE，
                       PROT_READ | PROT_WRITE，
                       MAP_FIXED | MAP_PRIVATE，
                       -1，0）;
如果（segmentad == MAP_FAILED）{PERROR（MMAP MAP_FIXED）;出口（EXIT_FAILURE）; }
 

这重新分配与 MAP_FIXED 将使用一些资源（如消耗一些交换空间）。

IIRC，在 SBCL 运行和垃圾回收使用这样的招数。

阅读也高级Linux编程并仔细的系统调用（2）以及相关的系统调用的特定手册页。

阅读也了解内存过量。这是一个Linux的功能，我不喜欢和一般禁用（如直通 PROC（5 ））。

顺便说一句，在 Linux内核是的免费软件。您可以从 kernel.org 下载源$ C ​​$ c和研究源$ C ​​$ C。你可以写一些实验code也。然后问另外一个更集中的问题，显示你的code和您的实验结果。

Since I read the Linux Programming Interface book (very good read) I discovered the file holes in Linux. Therefore using Unix file formats like Ext4 one can open a new file write 1000 bytes seek to the position of 1000.000.000 write another 1000 bytes and depending the block size of the file format end up with a file consuming just 2048 bytes for a block size of 1024 or 512.

So basically the file is created as a 1GB + 1000 bytes long file where only two blocks of real drive space are used.

Can I erase the middle of a file forcing the system to deallocate those blocks on the drive?

Is there an equivalent where I allocate (shared memory) with or without a file backing it where it has also holes that are just filled as the memory pages are written?

Would be nice to allocate 1GB shared memory but never utilize it fully until necessary just to avoid remapping if the shared memory block should grow.

解决方案

Can I erase the middle of a file forcing the system to deallocate those blocks on the drive?

You probably want the Linux specific fallocate(2); beware, it might not work on some filesystems (e.g. NFS, VFAT, ...), because some filesystems don't have holes. See also lseek(2) with SEEK_HOLE, posix_fadvise(2), madvise(2), memfd_create(2), etc...

Raw block devices (like a disk partition, or an USB key, or an SSD) don't have any holes (but you could mmap into them). Holes are a file system software artifact.

Would be nice to allocate 1GB shared memory but never utilize it

this is contradictory. If the memory is shared it is used (by the thing -generally another process- with which you share that memory). Read shm_overview(7) if you really want shared memory (and read carefully mmap(2)). Read more about virtual memory, address space, paging, MMUs, page faults, operating systems, kernels, mmap, demand paging, copy-on-write, memory map, ELF, sparse files ... Try also the cat /proc/$$/maps command in a terminal, and understand the output (see proc(5)...).

Perhaps you want to pre-allocate some address space range, and later really allocate the virtual memory. This is possible using Linux version of mmap(2).

To pre-allocate a gigabyte memory range, you'll first call mmap with MAP_NORESERVE

size_t onegiga = 1L<<30;
void* startad = mmap(NULL, onegiga, PROT_NONE, 
                MAP_ANONYMOUS|MAP_NORESERVE|MAP_SHARED,
                -1, 0);
if (startad==MAP_FAILED) { perror("mmap MAP_NORESERVE"); exit(EXIT_FAILURE); }
void* endad = (char*)startad + onegiga;

The MAP_NORESERVE does not consume a lot of resources (i.e. does not eat swap space, which is not reserved, hence the name of the flag). It is pre-allocating address space, in the sense that further mmap calls (without MAP_FIXED) won't give an address inside the returned range (unless you munmap some of it).

Later on, you can allocate some subsegment of that, in multiples of the page size (generally 4Kbytes), using MAP_FIXED inside the previous segment, e.g.

size_t segoff = 1024*1024; // or something else such that ....
assert (segoff >=0  && segoff < onegiga && segoff % sysconf(_SC_PAGESIZE)==0);
size_t segsize = 65536; // or something else such that ....
assert (segsize > 0 && segsize % sysconf(_SC_PAGESIZE)==0 
        && startad + segoffset + segsize < endad);
void* segmentad = mmap(startad + segoffset, segsize,
                       PROT_READ|PROT_WRITE, 
                       MAP_FIXED | MAP_PRIVATE, 
                       -1, 0);
if (segmentad == MAP_FAILED) { perror("mmap MAP_FIXED"); exit(EXIT_FAILURE); }

This re-allocation with MAP_FIXED will use some resources (e.g. consume some swap space).

IIRC, the SBCL runtime and garbage collection uses such tricks.

Read also Advanced Linux Programming and carefully syscalls(2) and the particular man pages of relevant system calls.

Read also about memory overcommitment. This is a Linux feature that I dislike and generally disable (e.g. thru proc(5)).

BTW, the Linux kernel is free software. You can download its source code from kernel.org and study the source code. And you can write some experimental code also. Then ask another more focused question showing your code and the results of your experiments.

这篇关于文件空洞和共享内存在Linux呢？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！