Julia-读取大文件的并行性 [英] Julia - Parallelism for Reading a Large file

问题描述

在Julia v1.1中，假设我有一个非常大的文本文件(30GB)，并且我希望并行处理(多线程)来读取每一行，我该怎么办?

In Julia v1.1, assume that I have a very large text file (30GB) and I want parallelism (multi-threads) to read eachline, how can I do ?

此代码是在检查

This code is an attempt to do this after checking Julia's documentation on multi-threading, but it's not working at all

open("pathtofile", "r") do file
    # Count number of lines in file
    seekend(file)
    fileSize = position(file)
    seekstart(file)

    # skip nseekchars first characters of file
    seek(file, nseekchars)

    # progress bar, because it's a HUGE file
    p = Progress(fileSize, 1, "Reading file...", 40)
    Threads.@threads for ln in eachline(file)
        # do something on ln
        u, v = map(x->parse(UInt32, x), split(ln))
        .... # other interesting things
        update!(p, position(file))
    end    
end

注意1:您需要using ProgressMeter(我希望我的代码在读取文件的同时显示进度条)

Note 1 : you need using ProgressMeter (I want my code to show a progress bar while parallelism the file reading)

注2:nseekchars是一个整数，我想在文件的开头跳过的字符数

Note 2 : nseekchars is an Int and the number of characters I want to skip in the beginning in my file

注意3:代码可以正常工作，但是在for循环旁边没有Threads.@threads宏的情况下不会进行并行处理

Note 3 : the code is working but doesn't do parellelism without Threads.@threads macro next to the for loop

推荐答案

为获得最佳I/O性能:

For the maximum I/O performance:

1. 并行化硬件-使用磁盘阵列而不是单个驱动器.尝试搜索突袭表现以获得许多出色的解释(或问一个单独的问题)

1. Parallelize the hardware - that is use disk arrays rather than a single drive. Try searching for raid performance for many excellent explanations (or ask a separate question)

使用Julia 内存映射机制

s = open("my_file.txt","r")
using Mmap
a = Mmap.mmap(s)

1. 具有内存映射后，并行进行处理.提防线程的错误共享 (取决于您的实际情况).

1. Once having the memory mapping, do the processing in parallel. Beware of false sharing for threads (depends on your actual scenario).

这篇关于Julia-读取大文件的并行性的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！