如何使用Pandas读取UTF-8文件? [英] How to read UTF-8 files with Pandas?

问题描述

我有一个带有twitter数据的UTF-8文件，我正在尝试将其读取到Python数据框中，但是我只能得到一个'object'类型而不是unicode字符串:

I have a UTF-8 file with twitter data and I am trying to read it into a Python data frame but I can only get an 'object' type instead of unicode strings:

# file 1459966468_324.csv
#1459966468_324.csv: UTF-8 Unicode English text
df = pd.read_csv('1459966468_324.csv', dtype={'text': unicode})
df.dtypes
text               object
Airline            object
name               object
retweet_count     float64
sentiment          object
tweet_location     object
dtype: object

使用Pandas将UTF-8数据读取并强制转换为unicode的正确方法是什么?

What is the right way of reading and coercing UTF-8 data into unicode with Pandas?

这不能解决问题:

df = pd.read_csv('1459966468_324.csv', encoding = 'utf8')
df.apply(lambda x: pd.lib.infer_dtype(x.values))

文本文件在这里: https://raw.githubusercontent.com/l1x/nlp/master/1459966468_324.csv

推荐答案

正如提到的其他海报一样，您可以尝试:

As the other poster mentioned, you might try:

df = pd.read_csv('1459966468_324.csv', encoding='utf8')

但是，当您打印dtypes时，这仍然会让您看到'object'.要确认它们是utf8，请在读取CSV之后尝试以下行:

However this could still leave you looking at 'object' when you print the dtypes. To confirm they are utf8, try this line after reading the CSV:

df.apply(lambda x: pd.lib.infer_dtype(x.values))

示例输出:

args            unicode
date         datetime64
host            unicode
kwargs          unicode
operation       unicode

这篇关于如何使用Pandas读取UTF-8文件?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！