将多个CSV文件读取到Python Pandas Dataframe中 [英] Reading Multiple CSV Files into Python Pandas Dataframe
问题描述
该问题背后的一般用例是将目标目录中的多个CSV日志文件读取到单个Python Pandas DataFrame中,以进行快速周转统计分析&图表.利用Pandas vs MySQL的想法是全天定期进行数据导入或附加+统计分析.
The general use case behind the question is to read multiple CSV log files from a target directory into a single Python Pandas DataFrame for quick turnaround statistical analysis & charting. The idea for utilizing Pandas vs MySQL is to conduct this data import or append + stat analysis periodically throughout the day.
以下脚本尝试将所有CSV(相同文件布局)文件读入单个Pandas数据框&添加与每个读取的文件关联的年份列.
The script below attempts to read all of the CSV (same file layout) files into a single Pandas dataframe & adds a year column associated with each file read.
# Assemble all of the data files into a single DataFrame & add a year field
# 2010 is the last available year
years = range(1880, 2011)
for year in years:
path ='C:\\Documents and Settings\\Foo\\My Documents\\pydata-book\\pydata-book-master`\\ch02\\names\\yob%d.txt' % year
frame = pd.read_csv(path, names=columns)
frame['year'] = year
pieces.append(frame)
# Concatenates everything into a single Dataframe
names = pd.concat(pieces, ignore_index=True)
# Expected row total should be 1690784
names
<class 'pandas.core.frame.DataFrame'>
Int64Index: 33838 entries, 0 to 33837
Data columns:
name 33838 non-null values
sex 33838 non-null values
births 33838 non-null values
year 33838 non-null values
dtypes: int64(2), object(2)
# Start aggregating the data at the year & gender level using groupby or pivot
total_births = names.pivot_table('births', rows='year', cols='sex', aggfunc=sum)
# Prints pivot table
total_births.tail()
Out[35]:
sex F M
year
2010 1759010 1898382
推荐答案
DataFrame实例上的append
方法的功能与列表实例上的append
方法的功能不同. Dataframe.append()
不会就地发生,而是返回一个新对象.
The append
method on an instance of a DataFrame does not function the same as the append
method on an instance of a list. Dataframe.append()
does not occur in-place and instead returns a new object.
years = range(1880, 2011)
names = pd.DataFrame()
for year in years:
path ='C:\\Documents and Settings\\Foo\\My Documents\\pydata-book\\pydata-book-master`\\ch02\\names\\yob%d.txt' % year
frame = pd.read_csv(path, names=columns)
frame['year'] = year
names = names.append(frame, ignore_index=True)
,或者您可以使用concat
:
years = range(1880, 2011)
names = pd.DataFrame()
for year in years:
path ='C:\\Documents and Settings\\Foo\\My Documents\\pydata-book\\pydata-book-master`\\ch02\\names\\yob%d.txt' % year
frame = pd.read_csv(path, names=columns)
frame['year'] = year
names = pd.concat(names, frame, ignore_index=True)
这篇关于将多个CSV文件读取到Python Pandas Dataframe中的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!