pandas 分割断层 [英] Pandas segmentation fault
本文介绍了 pandas 分割断层的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!
问题描述
由于内存不足,以下代码行未成功执行.
The following lines of code is not being successfully executed because of low memory.
import pandas as pd
import datetime as dt
u_cols=['remote_host', 'dummy1', 'dummy2', 'date', 'timezone', 'get', 'status', 'bytes_sent', 'dummy3', 'logline']
logfile='/var/log/apache2/error.log.1'
info = pd.read_csv(logfile, sep=' ', names=u_cols, low_memory=False)
In [5]: info = pd.read_csv(logfile, sep=' ', names=u_cols, low_memory=False)
Segmentation fault (core dumped)
# free -m
total used free shared buffers cached
Mem: 590 495 94 0 25 89
-/+ buffers/cache: 380 209
Swap: 0 0 0
如何正常退出? 此错误导致Django停止工作.
How do I exit gracefully? This error is causing Django to stop working.
>>> pd.show_versions()
INSTALLED VERSIONS
------------------
commit: None
python: 2.7.7.final.0
python-bits: 64
OS: Linux
OS-release: 3.2.0-31-virtual
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
pandas: 0.14.1
nose: 1.3.3
Cython: 0.20.1
numpy: 1.8.1
scipy: 0.14.0
statsmodels: 0.5.0
IPython: 2.1.0
sphinx: 1.2.2
patsy: 0.2.1
scikits.timeseries: None
dateutil: 1.5
pytz: 2014.3
bottleneck: None
tables: 3.1.1
numexpr: 2.3.1
matplotlib: 1.3.1
openpyxl: 1.8.5
xlrd: 0.9.3
xlwt: 0.7.5
xlsxwriter: 0.5.5
lxml: 3.3.5
bs4: 4.3.1
html5lib: None
httplib2: None
apiclient: None
rpy2: None
sqlalchemy: 0.9.4
pymysql: 0.6.2.None
psycopg2: 2.5.4 (dt dec pq3 ext)
您可以使用自己的错误日志进行尝试,也可以使用以下测试文件:
You can try this with your own error log or use a test file from:
http://testbyshantanu.s3.amazonaws.com/error.log.1
文件格式不正确,但不应强制python退出:)
The file is not correctly formatted but it should not force python to exit :)
推荐答案
需要在read_csv方法中添加"engine"参数,否则会产生分段错误.
need to add "engine" parameter to read_csv method or else get segmentation error.
In [5]: info = pd.read_csv(logfile, sep=' ', names=u_cols, engine='python' )
In [6]: info = pd.read_csv(logfile, sep=' ', names=u_cols )
Segmentation fault (core dumped)
这篇关于 pandas 分割断层的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!
查看全文