导入csv到pandas数据框时未读取所有行 [英] Not reading all rows while importing csv into pandas dataframe
问题描述
我正在尝试此处,并且很不幸,一个非常基本的步骤.我的有限的python知识必须为此负责. 我正在尝试通过执行以下命令,将数据集读取到熊猫数据框中. :
I am trying the kaggle challenge here, and unfortunately I am stuck at a very basic step. My limited python knowledge has to be blamed for this. I am trying to read the datasets into a pandas dataframe by executing following command:
test = pd.DataFrame.from_csv("C:/Name/DataMining/hillary/data/output/emails.csv")
问题在于,您将发现该文件具有超过300,000条记录,但我只读取了7945,21.
The problem is that this file as you would find out has over 300,000 records, but I am reading only 7945, 21.
print (test.shape)
(7945, 21)
现在,我已经仔细检查了文件,但找不到关于行号7945的特殊信息.任何可能导致这种情况的指针.看来情况很普通,我希望遇到这个错误的一些人能帮助我.
Now I have double checked the file and I cannot find anything special about line number 7945. Any pointers why this could be happening. Seems very ordinary situation, I hope some of you who have ran across this error can help me out.
推荐答案
我认为更好的方法是使用函数链接
I think better is use function read_csv with parameters quoting=csv.QUOTE_NONE
and error_bad_lines=False
. link
import pandas as pd
import csv
test = pd.read_csv("output/Emails.csv", quoting=csv.QUOTE_NONE, error_bad_lines=False)
print (test.shape)
#(381422, 22)
但是一些数据(有问题的)将被跳过.
But some data (problematic) will be skipped.
如果要跳过电子邮件正文数据,可以使用:
If you want skip emails body data, you can use:
import pandas as pd
import csv
test = pd.read_csv("output/Emails.csv", quoting=csv.QUOTE_NONE, sep=',', error_bad_lines=False, header=None,
names=["Id","DocNumber","MetadataSubject","MetadataTo","MetadataFrom","SenderPersonId","MetadataDateSent","MetadataDateReleased","MetadataPdfLink","MetadataCaseNumber","MetadataDocumentClass","ExtractedSubject","ExtractedTo","ExtractedFrom","ExtractedCc","ExtractedDateSent","ExtractedCaseNumber","ExtractedDocNumber","ExtractedDateReleased","ExtractedReleaseInPartOrFull","ExtractedBodyText","RawText"])
print (test.shape)
#delete row with NaN in column MetadataFrom
test = test.dropna(subset=['MetadataFrom'])
#delete headers in data
test = test[test.MetadataFrom != 'MetadataFrom']
这篇关于导入csv到pandas数据框时未读取所有行的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!