mremap(2)与HugeTLB更改虚拟地址? [英] mremap(2) with HugeTLB to change virtual address?

问题描述

Linux mremap(2)函数是否能够将从mmap()获得的HugeTLB的虚拟地址更改为新的固定虚拟地址?

(背景:我想根据获取的内存的物理地址重新映射虚拟地址.这是通过直接检查指针地址来有效地执行虚拟到物理地址的转换.我将使用DMA的内存到从用户空间.)

这似乎不适用于我的简单测试程序:

  #define _GNU_SOURCE#include< stdio.h>#include #include< stdint.h>#定义LARGE_PAGE_SIZE(1024 * 1024 * 1024)int main(){无效* p1;无效* p2;p1 = mmap(NULL，LARGE_PAGE_SIZE，PROT_READ | PROT_WRITE，MAP_SHARED | MAP_ANONYMOUS | MAP_HUGETLB | MAP_LOCKED，0，0);如果(p1 == MAP_FAILED){perror("mmap");返回1;}printf("p1 =％p \ n"，p1);p2 = mremap(p1，LARGE_PAGE_SIZE，LARGE_PAGE_SIZE，MREMAP_MAYMOVE | MREMAP_FIXED，(void*)(((uint64_t)p1) | 0x500000000000ULL));如果(p2 == MAP_FAILED){perror("mremap");返回1;}printf("p2 =％p \ n"，p2);} 

mremap() 失败导致 mmap() 成功:

  $ gcc -o mremap_hugetlb mremap_hugetlb.c&&须藤./mremap_hugetlbp1 = 0x2aaac0000000mremap:无效的参数 

请注意，新地址是根据原始mmap()获得的地址来计算的.这很重要.所需的地址无法提前知道，所以我不能简单地将MAP_FIXED传递给mmap().

我当前使用的解决方法是使mmap()支持文件，以便随后可以在固定地址再次mmap()并使用旧映射munmap().这不是最理想的，因为它要求我找到一个已挂载的ugeltlbfs文件系统，而且我不喜欢这种依赖关系的复杂性.

基于变通办法的当前代码: https://github.com/lukego/snabbswitch/blob/straightline/src/core/memory.c#L56

解决方案

现在看来您确实必须使用hugetlbfs.

除非我弄错了，否则问题发生在Linux内核中，因为 mm/mremap.c:vma_to_resize() ，其中 linux-内核和 linux-mm 邮件列表，以查看这是否应该/可以轻松修复的错误.但是，这对依赖当前(或更旧)内核的用户无济于事.

请记住，在文件描述符上使用 mmap()时，通常会使用不同的代码路径，因为每个文件系统都可以指定自己的 mmap 处理程序.对于hugetlbfs，代码位于 fs/hugetlbfs/inode.c:hugetlbfs_file_mmap() .而且，就像您说的那样，该代码路径对您来说似乎还行.

请注意，最好让用户配置hugetlbfs挂载点，而不是从/proc/mounts 中扫描一个，因为这样sysadmin可以配置多个hugetlbfs挂载点，每个针对服务器上运行的每个服务的不同配置.(我希望您的服务不需要以root用户身份运行.)

Is the Linux mremap(2) function able to change the virtual address of a HugeTLB obtained from mmap() to a new fixed virtual address?

(Background: I want to remap the virtual address based on the physical address of the memory I get. This is to efficiently perform virtual to physical address translations by inspecting pointer addresses directly. I will use the memory for DMA to hardware from userspace.)

This does not seem to work with my simple test program:

#define _GNU_SOURCE
#include <stdio.h>
#include <sys/mman.h>
#include <stdint.h>

#define LARGE_PAGE_SIZE (1024*1024*1024)

int main() {
  void *p1;
  void *p2;
  p1 = mmap(NULL, LARGE_PAGE_SIZE, PROT_READ|PROT_WRITE,
    MAP_SHARED|MAP_ANONYMOUS|MAP_HUGETLB|MAP_LOCKED,
    0, 0);
  if (p1 == MAP_FAILED) {
perror("mmap");
return 1;
  }
  printf("p1 = %p\n", p1);
  p2 = mremap(p1, LARGE_PAGE_SIZE, LARGE_PAGE_SIZE,
      MREMAP_MAYMOVE|MREMAP_FIXED,
      (void*)(((uint64_t)p1) | 0x500000000000ULL));
  if (p2 == MAP_FAILED) {
perror("mremap");
return 1;
  }
  printf("p2 = %p\n", p2);
}

The mmap() succeeds by the mremap() fails:

$ gcc -o mremap_hugetlb mremap_hugetlb.c && sudo ./mremap_hugetlb
p1 = 0x2aaac0000000
mremap: Invalid argument

Note that the new address is calculated from the one obtained by the original mmap(). This is significant. The desired address is not known ahead of time and so I can't simply pass MAP_FIXED to mmap().

The workaround I currently use is to make the mmap() file-backed so that I can then mmap() it again at a fixed address, and munmap() the old mapping. This is suboptimal because it requires me to find a mounted hugetlbfs filesystem and I don't like the complexity of that dependency.

Current code based on the workaround: https://github.com/lukego/snabbswitch/blob/straightline/src/core/memory.c#L56

解决方案

Right now it looks like you do have to use hugetlbfs.

Unless I'm mistaken, the problem occurs in the Linux kernel because mm/mremap.c:mremap_to() calls mm/mremap.c:vma_to_resize(), which fails with EINVAL for huge pages.

Perhaps the test is incorrect, or the function lacks code to handle huge pages correctly. I'm wondering if one should contact the linux-kernel and linux-mm mailing lists, to see if this is a bug that should/could be easily fixed. However, that won't help you with users relying on current (and older) kernels.

Remember that when using mmap() on a file descriptor, you usually use a different code path as each file system can specify their own mmap handler. For hugetlbfs, the code is in fs/hugetlbfs/inode.c:hugetlbfs_file_mmap(). And, like you said, that code path seems to work okay for you.

Note that it is best if you let the user configure the hugetlbfs mount point, instead of scanning one from /proc/mounts, as that way the sysadmin can configure multiple hugetlbfs mount points, each with different configuration, for each service running on the server. (I'm hoping your service does not require running as root.)

这篇关于mremap(2)与HugeTLB更改虚拟地址?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！