为什么我可以使用POSIX创建一个大于/dev/shm上安装的共享内存的共享内存? [英] Why I can create a shared memory bigger than the size mounted on /dev/shm using POSIX?

问题描述

我正在尝试在Ubuntu 16.04中使用共享内存IPC处理错误. 首先，我使用df -h在/dev/shm中检查了可用内存，具有500M可用空间，因此我快速编写了一些代码，以检查如果尝试创建大于安装大小的共享内存，会发生什么情况.代码如下(它已经被修改了好几次，所以我知道它不是很整洁):

I am trying to handle errors using shared memory IPC in a Ubuntu 16.04. First, I checked the available memory in /dev/shm using df -h, having 500M availables, so I coded something quickly in order to check what happens if I try to create a shared mem bigger than the mounted size. The code is the following (it has been modified several times so I know that is not very tidy):

#include <iostream>
#include <stdio.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <sys/ipc.h>
#include <sys/shm.h>
#include <sys/mman.h>
#include <fcntl.h>
#include <unistd.h>
#include <errno.h>
#include <cstring>
#include <stdint.h>
#include <stddef.h>
#include <cerrno>

//static const size_t size = 4000000L;
static const size_t size = 701000000L;
//static const size_t size = 999999999L;

extern int errno;

static int open(const char* name, int oflag, mode_t mode)
{
   int shm = -1;

   /* Create/Open a shared memory zone */
    shm = shm_open(name, oflag, mode);
    if(shm == -1)
    {/* Print error */
        std::cout << "!!!Error getting file descriptor while opening!!!" << std::endl;
        std::cout << "ERROR:"<< strerror(errno) << std::endl;
    }
   return shm;
}

static void write_shm(void *addr, size_t size)
{
    size_t i = 0;
    uint32_t *shm_index = (uint32_t *)addr;

    /* 4 bytes to be written in memory */
    const char *test = "DEAD";

    /* Maximum allowed memory address*/
    ptrdiff_t max = (size+(ptrdiff_t)addr);

    for (i = 0; (ptrdiff_t)(shm_index + i) < max; i++)
    {
        std:memcpy(&shm_index[i], (uint32_t*)test, sizeof(uint32_t));
    }
}
static int adjust (int fd, size_t size)
{
     std::cout<<__func__<<": The size of the shared memory is: "<<size<< std::endl;
     int result = ftruncate(fd, size);
     std::cout<<__func__<< "ftruncate return: " <<result<< std::endl;
     errno = 0;
     std::cout << "errno: "<< std::strerror(errno) <<std::endl;
     if (result)
     {/* Print error */;
        std::cout << "FUNCION!!!Error in ftruncate!!!" << std::endl;
     }
     return result;
}

int main()
{
    const char *name = "vardb";
    int fd = -1;
    int oflag = O_CREAT | O_EXCL | O_RDWR;
    mode_t mode = S_IRWXU | S_IRWXG | S_IRWXO; // POSIX 1003.1 (Realtime Extensions)
    size_t sizeToUse = (size/sizeof(uint32_t)* sizeof(uint32_t));

    /* Let's try to get a file descriptor related to the shared memory*/
    fd = open(name, oflag, mode);
    if (fd == -1)
        return fd;

    /* Adjust the size of the shared memory to the expected one */
    int result = adjust(fd, sizeToUse);
    if (result)
        return -1;

    int prot = PROT_READ | PROT_WRITE;
    int flags = MAP_SHARED;

    /* Map the memory */
    void *addr = mmap(NULL, size, prot, flags, fd, 0);
    std::cout<<__func__<< "mmap return: " <<*(int *)addr<< std::endl;
    std::cout<<__func__<< "mmap mapped to this address: " <<addr<< std::endl;
    errno = 0;
    std::cout << "mmap errno: "<< std::strerror(errno) <<std::endl;

    struct stat fileStat;
    if(fstat(fd, &fileStat) < 0)    
        return 1;

    std::cout<<__func__<< "File Size: " <<fileStat.st_size<<" bytes"<<std::endl;

    /* Write all the shared memory previously reserved for the test */
    write_shm(addr, sizeToUse);


    /* Release the memory */
    munmap(addr, size);

    return 0;
}

我不会取消链接共享内存以进行十六进制转储，因此需要在重新启动程序之前手动将其删除.

I do not unlink the shared mem in order to hexdump it, so this requires to remove it manually before relaunching the program.

好吧，我发出的是在创建比/dev/shm安装大小更大的共享内存时，我没有收到任何错误...显然，当我尝试写出一个错误时，我遇到了一个总线错误.可用的内存范围，但我需要控制共享内存的创建...我无法理解系统如何让我在不报告任何错误的情况下创建类似的内容.

Well, what I am issuing is that I do not get any error while I am creating a bigger shared mem than the /dev/shm mounted size... Obviously, I get a Bus Error as I try to write out of the available range of memory but I need to control the creation of the shared memory... I cannot understand how the system let me create something like this without reporting me any error.

谢谢.

最诚挚的问候，

伊万.

推荐答案

简短(且不令人满意的答案):如果/dev/shm上的空间不足，则不能强制shm_open失败. (您可以使用setrlimit来修改RLIMIT_FSIZE来显式设置进程文件大小限制，以强制其失败，但这是全局设置，不适用于单个文件系统，因此几乎可以肯定这不是您想要的.)

The short (and unsatisfying answer): you can't force shm_open to fail if there is insufficient space on /dev/shm. (You can force it to fail by explicitly setting the process filesize limit using setrlimit to modify RLIMIT_FSIZE, but that's a global setting not applicable to a single filesystem, so it is almost certainly not what you want it to do.)

当Posix标准化共享内存时，考虑了各种实现选项，并且尝试了使实现具有相当大的灵活性，只要它不使接口复杂化即可.特别是，许多Unix实现已经具有将文件对象直接映射到进程内存以及基于内存的文件系统的机制，这种组合非常适合共享内存的实现:

When Posix standardised shared memory, various implementation options were considered and an attempt was made to allow considerable flexibility to implementations as long as it did not complicate the interface. In particular, many Unix implementations already had mechanisms for mapping file objects directly to process memory, and for memory-based filesystems, a combination ideally suited to the implementation of shared memory:

简单的共享内存显然是更通用的文件映射功能的特例.另外，文件映射接口有相对广泛的协议和实现.在这些系统中，可以使用相同的映射接口来映射许多不同类型的对象(例如，文件，内存，设备等).这种方法既可以最大程度地减少接口扩散，又可以使用映射接口最大程度地提高程序的通用性. (摘自 Posix原理:映射的文件功能)

特别地，……以上要求不排除:使用实际文件系统上的实际文件来实现可共享存储对象." ( Posix原理:共享内存对象).尽管我不相信Linux库会这样做，但Posix甚至允许shm_open()被实现为包装普通open()调用的宏.对于仅将共享内存映射到文件系统的实现(例如Linux)，不需要对 ftruncate() 系统界面.

In particular, "…the above requirements do not preclude: [t]he sharable memory object from being implemented using actual files on an actual file system." (Posix rationale: Shared memory objects). Although I don't believe the Linux library does this, Posix even allows shm_open() to be implemented as a macro wrapping an ordinary open() call; in the case of implementations (like Linux) which simply map shared memory onto the filesystem, nothing requires special handling of the ftruncate() system interface.

重要的是要突出ftruncate()通话的一个方面(添加了重点):

It's important to highlight one aspect of the ftruncate() call (emphasis added):

如果文件大小增加，则扩展区域应显示为，好像它是零填充的.

If the file size is increased, the extended area shall appear as if it were zero-filled.

许多文件系统允许稀疏文件".在稀疏文件中，完全填充零的文件块不会直接映射到物理存储；如果应用程序读取这些块之一，则会收到零页.如果修改了一个块并将其提交到磁盘，则文件系统(并且仅在此之后)才为该块分配存储. [注1]

Many filesystems allow for "sparse files". In a sparse file, file blocks entirely filled with zeros are simply not mapped onto physical storage; if an application reads one of these blocks, it receives a page of zeros. If a block is modified and committed to disk, then -- and only then -- does the filesystem allocate storage for the block. [Note 1]

零填充块的惰性分配意味着，在ftruncate()扩展文件的情况下，它仅需要更新文件的元数据，即可非常快速地返回.除非所需的大小超出进程的文件大小限制(或在文件系统没有为文件大小使用足够大的整数类型的文件系统的情况下，文件系统限制)，否则ftruncate()不会产生任何错误.当无法分配物理存储(如果是内存映射文件，则无法分配专用的内存缓冲区)时，将发生此错误.

Lazy allocation of zero-filled blocks means that in the case where ftruncate() is expanding a file, it only needs to update the file's metadata, allowing it to return very rapidly. Unless the desired size exceeds the process's file size limit (or the filesystem limit in the case of filesystems which don't use large enough integer types for file sizes), no error will be produced by ftruncate(). The error will occur when it becomes impossible to allocate physical storage (or a dedicate memory buffer, in the case of a memory-mapped file).

这完全与Linux乐观的内存分配方法一致:mmap总是成功(只要地址空间可用)，并且仅在实际使用内存时才指出错误. (但是这种共享内存实现的特定方法并不限于使用乐观内存分配的方法.)

This is entirely consistent with Linux's optimistic approach to memory allocation: mmap always succeeds (as long as address space is available) and the error is only noted when the memory is actually used. (But this particular approach to shared memory implementation is not limited to those using optimistic memory allocation.)

1. 我曾经通过在软盘上存储2GB的文件来演示这一点，但我想今天很多读者甚至都不知道软盘是什么，而实际容量却更少.

这篇关于为什么我可以使用POSIX创建一个大于/dev/shm上安装的共享内存的共享内存?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！