我该如何分配由1GB大页面Linux内核模块支持的DMA缓冲区？ [英] How do I allocate a DMA buffer backed by 1GB HugePages in a linux kernel module?

问题描述

我试图分配一个DMA缓冲区的HPC工作量。它需要的缓冲空间64GB。在计算之间的一些数据被卸载到一个PCIe卡。而不是将数据复制到一组由pci_alloc_consistent给出极小4MB缓存的，我想只是创建64 1GB缓存，1GB通过支持大页面。该系统具有的RAM 140GB所以这应该是身体可能。

I'm trying to allocate a DMA buffer for a HPC workload. It requires 64GB of buffer space. In between computation, some data is offloaded to a PCIe card. Rather than copy data into a bunch of dinky 4MB buffers given by pci_alloc_consistent, I would like to just create 64 1GB buffers, backed by 1GB HugePages. The system has 140GB of RAM so this should be physically possible.

一些背景信息：
内核版本：CentOS的6.4 / 2.6.32-358.el6.x86_64
内核引导选项：hugepagesz =1克大页面= 64 default_hugepagesz =1克

Some background info: kernel version: CentOS 6.4 / 2.6.32-358.el6.x86_64 kernel boot options: hugepagesz=1g hugepages=64 default_hugepagesz=1g

中的/ proc / meminfo中的相关部分：
AnonHugePages：0 KB
HugePages_Total：64
HugePages_Free：64
HugePages_Rsvd：0
HugePages_Surp：0
Hugepagesize：1048576 KB
DirectMap4k：848 KB
DirectMap2M：2062336 KB
DirectMap1G：132120576 KB

relevant portion of /proc/meminfo: AnonHugePages: 0 kB HugePages_Total: 64 HugePages_Free: 64 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 1048576 kB DirectMap4k: 848 kB DirectMap2M: 2062336 kB DirectMap1G: 132120576 kB

我可以装入-t hugetlbfs的为nodev到/ mnt /大页面。 CONFIG_HUGETLB_PAGE是真实的。 MAP_HUGETLB定义

I can mount -t hugetlbfs nodev /mnt/hugepages. CONFIG_HUGETLB_PAGE is true. MAP_HUGETLB is defined.

我已经阅读使用libhugetlbfs调用get_huge_pages（）在用户空间的一些信息，但理想的这个缓冲区将在内核空间进行分配。我打过电话do_mmap（）与MAP_HUGETLB，但它似乎没有改变的自由大页面的数量，所以我不认为它实际上是支持巨大的页面MMAP。

I have read some info on using libhugetlbfs to call get_huge_pages() in user space, but ideally this buffer would be allocated in kernel space. I tried calling do_mmap() with MAP_HUGETLB but it didn't seem to change the number of free hugepages, so I don't think it was actually backing the mmap with huge pages.

所以，我想我在获得，是否有的的办法，我可以映射在内核空间的缓冲区到1GB HugePage，还是有在用户空间做？或者，如果任何人的任何其他方式知道我能得到一个巨大的（1-64GB）的连续物理内存可以作为一个内核缓冲区量？

So I guess what I'm getting at, is there any way I can map a buffer to a 1GB HugePage in kernel space, or does it have to be done in user space? Or if anyone knows of any other way I can get an immense (1-64GB) amount of contiguous physical memory available as a kernel buffer?

推荐答案

这是不常用的内核空间完成的，因此没有太多的例子。

This is not commonly done in the kernel space, so not too many examples.

就像任何其他页面上，大内存页被分配alloc_pages，以调：

Just like any other page, huge pages are allocated with alloc_pages, to the tune:

struct page *p = alloc_pages(GFP_TRANSHUGE, HPAGE_PMD_ORDER);

HPAGE_PMD_ORDER是宏，在正常的网页来定义单个巨大页的顺序。以上意味着透明大页面在内核中启用。

HPAGE_PMD_ORDER is a macro, defining an order of a single huge page in terms of normal pages. The above implies that transparent huge pages are enabled in kernel.

然后就可以进行映射以通常的方式与KMAP所获得的页面指针（）。

Then you can proceed mapping the obtained page pointer in the usual fashion with kmap().

免责声明：我从来没有尝试过自己，所以你可能需要做一些试验左右。有一件事要检查的是：HPAGE_PMD_SHIFT重新presents一个较小的庞大的页面顺序。如果你想利用这些巨头1GB的页面，你可能需要尝试不同的顺序，可能PUD_SHIFT - PAGE_SHIFT

Disclaimer: I never tried it myself, so you may have to do some experimenting around. One thing to check for is this: HPAGE_PMD_SHIFT represents an order of a smaller "huge" page. If you want to use those giant 1GB pages, you will probably need to try a different order, probably PUD_SHIFT - PAGE_SHIFT.

这篇关于我该如何分配由1GB大页面Linux内核模块支持的DMA缓冲区？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！