Julia - 读取大文件的并行性 [英] Julia - Parallelism for Reading a Large file

问题描述

In Julia v1.1, assume that I have a very large text file (30GB) and I want parallelism (multi-threads) to read eachline, how can I do ?

This code is an attempt to do this after checking Julia's documentation on multi-threading, but it's not working at all

open("pathtofile", "r") do file
    # Count number of lines in file
    seekend(file)
    fileSize = position(file)
    seekstart(file)

    # skip nseekchars first characters of file
    seek(file, nseekchars)

    # progress bar, because it's a HUGE file
    p = Progress(fileSize, 1, "Reading file...", 40)
    Threads.@threads for ln in eachline(file)
        # do something on ln
        u, v = map(x->parse(UInt32, x), split(ln))
        .... # other interesting things
        update!(p, position(file))
    end    
end

Note 1 : you need using ProgressMeter (I want my code to show a progress bar while parallelism the file reading)

Note 2 : nseekchars is an Int and the number of characters I want to skip in the beginning in my file

Note 3 : the code is working but doesn't do parellelism without Threads.@threads macro next to the for loop

解决方案

For the maximum I/O performance:

1. Parallelize the hardware - that is use disk arrays rather than a single drive. Try searching for raid performance for many excellent explanations (or ask a separate question)

2. Use the Julia memory mapping mechanism

s = open("my_file.txt","r")
using Mmap
a = Mmap.mmap(s)

3. Once having the memory mapping, do the processing in parallel. Beware of false sharing for threads (depends on your actual scenario).

这篇关于Julia - 读取大文件的并行性的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！