与Python相比，在Julia中读取csv速度较慢 [英] reading csv in Julia is slow compared to Python
问题描述
在Julia中读取大文本/csv文件需要花费很长时间.这是读取大小为486.6 MB，具有153895行和644列的文件的时间.
reading large text / csv files in Julia takes a long time compared to Python. Here are the times to read a file whose size is 486.6 MB and has 153895 rows and 644 columns.
python 3.3示例
python 3.3 example
import pandas as pd
import time
start=time.time()
myData=pd.read_csv("C:\\myFile.txt",sep="|",header=None,low_memory=False)
print(time.time()-start)
Output: 19.90
R 3.0.2示例
system.time(myData<-read.delim("C:/myFile.txt",sep="|",header=F,
stringsAsFactors=F,na.strings=""))
Output:
User System Elapsed
181.13 1.07 182.32
Julia 0.2.0(Julia Studio 0.4.4)示例#1
Julia 0.2.0 (Julia Studio 0.4.4) example # 1
using DataFrames
timing = @time myData = readtable("C:/myFile.txt",separator='|',header=false)
Output:
elapsed time: 80.35 seconds (10319624244 bytes allocated)
Julia 0.2.0(Julia Studio 0.4.4)示例#2
Julia 0.2.0 (Julia Studio 0.4.4) example # 2
timing = @time myData = readdlm("C:/myFile.txt",'|',header=false)
Output:
elapsed time: 65.96 seconds (9087413564 bytes allocated)
-
Julia比R更快，但与Python相比却相当慢.我可以采取什么其他措施来加快读取大文本文件的速度?
Julia is faster than R, but quite slow compared to Python. What can I do differently to speed up reading a large text file?
一个单独的问题是内存大小是Julia文件中硬盘文件大小的18倍，而python文件大小只有2.5倍.在Matlab中，我发现大型文件的内存使用效率最高，它是硬盘文件大小的2倍.茱莉亚(Julia)内存中的文件大的任何特殊原因?
a separate issue is the size in memory is 18 x size of hard disk file size in Julia, but only 2.5 x size for python. in Matlab, which I have found to be most memory efficient for large files, it is 2 x size of hard disk file size. Any particular reason for the large file size in memory in Julia?
推荐答案
最好的答案可能是我不像Wes那样优秀.
The best answer is probably that I'm not as a good a programmer as Wes.
通常，DataFrames中的代码与Pandas中的代码相比没有得到很好的优化.我相信我们可以赶上，但是这需要一些时间，因为首先需要实现许多基本功能.由于在Julia中需要构建的内容太多，因此我倾向于将工作分为三个部分:(1)构建任何版本，(2)构建正确的版本，(3)构建快速正确的版本.对于我所做的工作，Julia通常不提供任何版本的基本功能，因此我的工作重点是(1)和(2).随着我需要构建更多工具，集中精力于性能将变得更加容易.
In general, the code in DataFrames is much less well-optimized than the code in Pandas. I'm confident that we can catch up, but it will take some time as there's a lot of basic functionality that we need to implement first. Since there's so much that needs to be built in Julia, I tend to focus on doing things in three parts: (1) build any version, (2) build a correct version, (3) built a fast, correct version. For the work I do, Julia often doesn't offer any versions of essential functionality, so my work gets focused on (1) and (2). As more of the tools I need get built, it'll be easier to focus on performance.
关于内存使用，我认为答案是，在解析表格数据时，我们使用了一组数据结构，这些数据结构的效率远低于Pandas所使用的数据结构.如果我更好地了解了Pandas的内部，我可以列出一些效率较低的地方，但是现在我只能推测一个明显的失败是我们正在将整个数据集读取到内存中，而不是从磁盘上抓取块.当然可以避免这种情况，这样做存在很多问题.这只是时间问题.
As for memory usage, I think the answer is that we use a set of data structures when parsing tabular data that's much less efficient than those used by Pandas. If I knew the internals of Pandas better, I could list off places where we're less efficient, but for now I'll just speculate that one obvious failing is that we're reading the whole dataset into memory rather than grabbing chunks from disk. This certainly can be avoided and there are issues open for doing so. It's just a matter of time.
请注意，readtable代码非常易于阅读.使readtable更快的最确定的方法是鞭打Julia探查器，并开始修复它发现的性能缺陷.
On that note, the readtable code is fairly easy to read. The most certain way to get readtable to be faster is to whip out the Julia profiler and start fixing the performance flaws it uncovers.
这篇关于与Python相比，在Julia中读取csv速度较慢的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
