强制JVM在没有页面缓存的情况下执行所有IO（例如O_DIRECT） [英] Force JVM to do all IO without page cache (e.g. O_DIRECT)

问题描述

我正在用Java编写的应用程序的一些基准测试。对于实验来说非常重要的是结果不受页面缓存的影响（我正在使用linux）

I'm doing some benchmarks of an application written in Java. It is very important for the experiments that the results are not influenced by the page cache (I'm using linux)

因此避免页面缓存的最佳方法是使用每当打开文件时O_DIRECT。因此，我更改了jre源代码中的相应代码。

So the best way to avoid the page cache is using O_DIRECT whenever a file is opened. I therefore changed the respective code in the sourcecode of the jre.

我的方法适用于通过 FileOutputStream （例如写作），但它不适用于 FileInputStream （例如阅读）。

My approach works perfectly for everything that goes through the FileOutputStream (e.g. writing), but it does not work for FileInputStream (e.g. reading).

添加O_DIRECT时对于 FileInputStream 的开放调用，JVM无法加载任何类：

When adding O_DIRECT to the open-call of FileInputStream, the JVM is unable to load any classes:

Error: Could not find or load main class perf.TestDirectIO

此错误不是classpath问题，因为我可以通过使用unhackedJVM来修复它。

This error is not a classpath issue, since I can fix it just by using a "unhacked" JVM.

因此打开文件似乎存在问题。

So there seems to be an issue with opening files.

我对如何解决这个问题的任何建议感到非常高兴。

I'm very happy about any advice on how to fix the issue.

如果有人想做类似的事，我会在我的博客中记录了整个黑客。

If anyone ever wants to do something similar, I've documented the whole hack in my blog.

作为参考，这些是我对JVM代码所做的更改：

As a reference, these are the changes on the JVM code I did so:

jdk / src / share / native / java / io / FileInputStream.c ：

 @@ -58,7 +60,8 @@
 JNIEXPORT void JNICALL
 Java_java_io_FileInputStream_open(JNIEnv *env, jobject this, jstring path) {
-    fileOpen(env, this, path, fis_fd, O_RDONLY);
+    fileOpen(env, this, path, fis_fd, O_RDONLY | O_DIRECT); // this is the change that causes all the problems
 }

此更改有效：
jdk / src / solaris / native / java / io / FileOutputStream_md.c ：

@@ -55,8 +55,10 @@
 JNIEXPORT void JNICALL
 Java_java_io_FileOutputStream_open(JNIEnv *env, jobject this,
                                    jstring path, jboolean append) {
     fileOpen(env, this, path, fos_fd,
-             O_WRONLY | O_CREAT | (append ? O_APPEND : O_TRUNC));
+             O_WRONLY | O_DIRECT | O_CREAT | (append ? O_APPEND : O_TRUNC));
 }

我还改变了热点jre以确保内存对齐（这是一个要求O_DIRECT）
hotspot / src / share / vm / runtime / os.cpp ：

I also changed the hotspot jre to ensure that the memory is aligned (thats a requirement for O_DIRECT) hotspot/src/share/vm/runtime/os.cpp:

+# include <mm_malloc.h>
...
-  u_char* ptr = (u_char*)::malloc(size + space_before + space_after);
+  u_char* ptr = (u_char*)::_mm_malloc(size + space_before + space_after,512);

推荐答案

  "The thing that has always disturbed me about O_DIRECT is that the
   whole interface is just stupid, and was probably designed by a deranged
   monkey on some serious mind-controlling substances  [*]."

[*]换句话说，它是甲骨文。

[*] In other words, it's an Oracleism.

- 2002年5月11日，来自Transmeta的 Linus Torvalds

-- Linus Torvalds from Transmeta, 11 May 2002

检查 man 2 open ：

O_DIRECT

O_DIRECT

O_DIRECT标志可能对用户空间缓冲区的长度和地址
施加对齐限制I / O的文件偏移量。在Linux对齐中，
的限制因文件系统和内核版本而异....

The O_DIRECT flag may impose alignment restrictions on the length and address of userspace buffers and the file offset of I/Os. In Linux alignment restrictions vary by file system and kernel version ....

在Linux 2.4下，传输大小，以及用户缓冲区的对齐方式和
文件偏移量必须都是文件
系统的逻辑块大小的倍数。在Linux 2.6下，对齐到512字节边界就足够了。
....

Under Linux 2.4, transfer sizes, and the alignment of the user buffer and the file offset must all be multiples of the logical block size of the file system. Under Linux 2.6, alignment to 512-byte boundaries suffices. ....

总之，O_DIRECT是一个潜在的强大工具，应谨慎使用
。建议应用程序将O_DIRECT用作
性能选项，默认情况下禁用。

In summary, O_DIRECT is a potentially powerful tool that should be used with caution. It is recommended that applications treat use of O_DIRECT as a performance option which is disabled by default.

我认为，那里是JRE（类加载器）中FileInputStream的一些用法，它具有偏移量或大小未与512字节对齐的读取。 （对于高级格式，最小对齐可能更大，甚至是4096字节，或者是一个4K页面。）

I think, there are some usages of FileInputStream in the JRE (classloader) which has reads with offsets or sizes not aligned to 512 bytes. (For Advanced Format the minimal alignment may be bigger, even 4096 bytes, or one 4K page.)

未对齐偏移的内核行为是灰色区域，有些信息在这里： RFC：澄清直接I / O语义，Theodore Ts'o，tytso @ mit，LWN，2009

The behaviour of kernel for unaligned offsets is the grey zone, some info is here: RFC: Clarifying Direct I/O Semantics, Theodore Ts'o, tytso@mit, LWN, 2009

其他有趣的讨论是这里： Linux：使用O_DIRECT访问文件（kerneltrap，2007）

Other interesting discussion is here: Linux: Accessing Files With O_DIRECT (kerneltrap, 2007)

嗯，当DIRECT出现故障时，似乎应该回退到缓冲的I / O.使用DIRECT的所有IO操作都是同步的。可能是一些DMA效应？或者 O_DIRECT 和 mmap 的组合？

Hmm, seems like there should be fallback to buffered I/O when something with DIRECT fails. All IO operations with DIRECT are synchronous. May be some DMA effects? Or combination of O_DIRECT and mmap?

更新：

感谢strace输出。这是错误（ grep O_DIRECT ，然后检查文件描述符操作）：

Thanks for strace output. Here is the error (grep O_DIRECT, then check file descriptor operations):

28290 open("...pact/perf/TestDirectIO.class", O_RDONLY|O_DIRECT) = 11
28290 fstat(11, {st_mode=S_IFREG|0644, st_size=2340, ...}) = 0
28290 fcntl(11, F_GETFD)                = 0
28290 fcntl(11, F_SETFD, FD_CLOEXEC)    = 0
...skip
28290 stat("...pact/perf/TestDirectIO.class", {st_mode=S_IFREG|0644, st_size=2340, ...}) = 0
...skip
28290 read(11, "\312\376\272\276\0\0\0003\0\215\n\0-\0D\t\0E\0F\7\0G\n\0\3\0D\10\0H\n"..., 1024) = 1024
28290 read(11, 0x7f1d76d23a00, 1316)    = -1 EINVAL (Invalid argument)

未对齐的读取大小会导致 EINVAL 错误。你的类文件长2340字节，它是1024 + 1316字节，没有对齐。

Unaligned read size results in EINVAL error. Your classfile is 2340 bytes long, it is 1024+1316 bytes, which is not aligned.

这篇关于强制JVM在没有页面缓存的情况下执行所有IO（例如O_DIRECT）的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！