用 pandas 读取以空格分隔的数据 [英] Read Space-separated Data with Pandas
问题描述
我以前用numpy.loadtxt()
读取数据.但是,最近我在 SO 中发现,pandas.read_csv()
的速度要快得多.
I used to read my data with numpy.loadtxt()
. However, lately I found out in SO, that pandas.read_csv()
is much more faster.
要读取这些数据,请使用:
To read these data I use:
pd.read_csv(filename, sep=' ',header=None)
我现在遇到的问题是,在我的情况下,分隔符可以从一个空格, x 空格到一个制表符都不同.
The problem that I encounter right now is that in my case the separator can differ from one space, x spaces to even a tab.
这是我的数据的样子:
56.00 101.85 52.40 101.85 56.000000 101.850000 1
56.00 100.74 50.60 100.74 56.000000 100.740000 2
56.00 100.74 52.10 100.74 56.000000 100.740000 3
56.00 102.96 52.40 102.96 56.000000 102.960000 4
56.00 100.74 55.40 100.74 56.000000 100.740000 5
结果如下:
0 1 2 3 4 5 6 7 8
0 56 NaN NaN 101.85 52.4 101.85 56 101.85 1
1 56 100.74 50.6 100.74 56.0 100.74 2 NaN NaN
2 56 100.74 52.1 100.74 56.0 100.74 3 NaN NaN
3 56 102.96 52.4 102.96 56.0 102.96 4 NaN NaN
4 56 100.74 55.4 100.74 56.0 100.74 5 NaN NaN
我必须指定我的数据> 100 MB.因此,我无法预处理数据或先清除它们. 任何想法如何解决此问题?
I have to specify that my data are >100 MB. So I can not preprocess the data or clean them first. Any ideas how to get this fixed?
推荐答案
您的原始行:
pd.read_csv(filename, sep=' ',header=None)
将分隔符指定为单个空格,因为您的csvs可以包含空格或制表符,因此您可以将正则表达式传递给sep
参数,如下所示:
was specifying the separator as a single space, because your csvs can have spaces or tabs you can pass a regular expression to the sep
param like so:
pd.read_csv(filename, sep='\s+',header=None)
这将分隔符定义为一个或多个空白,并且有一个方便的备忘单,其中列出了正则表达式.
This defines separator as being one single white space or more, there is a handy cheatsheet that lists regular expressions.
这篇关于用 pandas 读取以空格分隔的数据的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!