为什么(ftruncate + mmap + memcpy)比(写入)更快? [英] Why (ftruncate+mmap+memcpy) is faster than (write)?

问题描述

我发现了另一种写数据的方式，它比普通的unix write 函数要快.

I found a different way to write data, which is faster than normal unix write function.

首先，将文件 ftruncate 调整到所需的长度，然后 mmap 该文件块，最后使用 memcpy 刷新文件内容.我将在下面给出示例代码.

Firstly, ftruncate the file to the length we need, then mmap this block of file, finally, using memcpy to flush the file content. I will give the example code below.

如我所知，mmap可以通过忽略页面缓存来加速文件加载到进程地址空间.但是，我不知道为什么它可以加快写入速度.

As I known, mmap can load the file into the process address space, accelerating by ignoring the page cache. BUT, I don't have any idea why it can fast up the writing speed.

我写错了测试用例还是一种优化技巧?

Whether I write a wrong test case or it can be a kind of opti trick?

这是测试代码.它的某些内容是用ObjC编写的，但没关系. WCTTicker 只是使用 gettimeofday 的统计信息类.

Here is the test code. Some of its written in ObjC, but no matter. WCTTicker is just a statistics class using gettimeofday.

//find a dir to test
NSString* document = NSSearchPathForDirectoriesInDomains(NSDocumentDirectory, NSUserDomainMask, YES)[0];
NSString* dir = [document stringByAppendingPathComponent:@"testDir"];

//remove all existing test
if ([[NSFileManager defaultManager] fileExistsAtPath:dir]) {
    if (![[NSFileManager defaultManager] removeItemAtPath:dir error:nil]) {
        NSLog(@"fail to remove dir");
        return;
    }
}
//create dir to test
if (![[NSFileManager defaultManager] createDirectoryAtPath:dir withIntermediateDirectories:YES attributes:nil error:nil]) {
    NSLog(@"fail to create dir");
}

//pre-alloc memory
const int length = 10000000;
const int count = 100;
char* mem = (char*)malloc(length);
memset(mem, 'T', length);

{
    //start testing mmap
    // ftruncate && mmap(private) &&memcpy
    NSString* mmapFileFormat = [dir stringByAppendingPathComponent:@"privateMmapFile%d"];
    [WCTTicker tick];
    for (int i = 0; i < count; i++) {
        NSString* path = [[NSString alloc] initWithFormat:mmapFileFormat, i];
        int fd = open(path.UTF8String, O_CREAT | O_RDWR, S_IRWXG | S_IRWXU | S_IRWXO);
        if (fd<0) {
            NSLog(@"fail to open");
        }
        int rc = ftruncate(fd, length);
        if (rc<0) {
            NSLog(@"fail to truncate");
        }
        char* map = (char*)mmap(NULL, length, PROT_WRITE | PROT_READ, MAP_PRIVATE, fd, 0);
        if (!map) {
            NSLog(@"fail to mmap");
        }
        memcpy(map, mem, length);
        close(fd);
    }
    [WCTTicker stop];
}

{
    //start testing write
    // normal write
    NSString* writeFileFormat = [dir stringByAppendingPathComponent:@"writeFile%d"];
    [WCTTicker tick];
    for (int i = 0; i < count; i++) {
        NSString* path = [[NSString alloc] initWithFormat:writeFileFormat, i];
        int fd = open(path.UTF8String, O_CREAT | O_RDWR, S_IRWXG | S_IRWXU | S_IRWXO);
        if (fd<0) {
            NSLog(@"fail to open");
        }
        int written = (int)write(fd, mem, length);
        if (written!=length) {
            NSLog(@"fail to write");
        }
        close(fd);
    }
    [WCTTicker stop];
}

{
    //start testing mmap
    // ftruncate && mmap(shared) &&memcpy
    NSString* mmapFileFormat = [dir stringByAppendingPathComponent:@"sharedMmapFile%d"];
    [WCTTicker tick];
    for (int i = 0; i < count; i++) {
        NSString* path = [[NSString alloc] initWithFormat:mmapFileFormat, i];
        int fd = open(path.UTF8String, O_CREAT | O_RDWR, S_IRWXG | S_IRWXU | S_IRWXO);
        if (fd<0) {
            NSLog(@"fail to open");
        }
        int rc = ftruncate(fd, length);
        if (rc<0) {
            NSLog(@"fail to truncate");
        }
        char* map = (char*)mmap(NULL, length, PROT_WRITE | PROT_READ, MAP_SHARED, fd, 0);
        if (!map) {
            NSLog(@"fail to mmap");
        }
        memcpy(map, mem, length);
        close(fd);
    }
    [WCTTicker stop];
}

这是测试结果:

2016-07-05 11:44:08.425 TestCaseiOS[4092:1070240] 
0: 1467690246.689788, info: (null)
1: 1467690248.419790, cost 1.730002, info: (null)
2016-07-05 11:44:14.126 TestCaseiOS[4092:1070240] 
0: 1467690248.427097, info: (null)
1: 1467690254.126590, cost 5.699493, info: (null)
2016-07-05 11:44:14.814 TestCaseiOS[4092:1070240] 
0: 1467690254.126812, info: (null)
1: 1467690254.813698, cost 0.686886, info: (null)

推荐答案

您有 mmap()，而没有相应的 munmap()

从 mmap 手册页(Linux)

From mmap manual page (linux)

MAP_SHARED共享此映射.映射的更新是可见的映射到该文件的其他进程，并一直传递到基础文件.直到msync(2)才可能实际更新文件或munmap()被调用.

MAP_SHARED Share this mapping. Updates to the mapping are visible to other processes that map this file, and are carried through to the underlying file. The file may not actually be updated until msync(2) or munmap() is called.

也许您应该再次运行测试，以便调用 munmap :

Perhaps you should run your tests again so that there is a call to munmap:

char* map = (char*)mmap(NULL, length, PROT_WRITE | PROT_READ, MAP_SHARED, fd, 0);
if (!map) {
   NSLog(@"fail to mmap");
}
memcpy(map, mem, length);
munmap(map, length);
close(fd);

这篇关于为什么(ftruncate + mmap + memcpy)比(写入)更快?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！