如何使用 Pandas 读取 UTF-8 文件? [英] How to read UTF-8 files with Pandas?

问题描述

我有一个带有 twitter 数据的 UTF-8 文件，我试图将它读入 Python 数据框，但我只能得到一个对象"类型而不是 unicode 字符串:

I have a UTF-8 file with twitter data and I am trying to read it into a Python data frame but I can only get an 'object' type instead of unicode strings:

# file 1459966468_324.csv
#1459966468_324.csv: UTF-8 Unicode English text
df = pd.read_csv('1459966468_324.csv', dtype={'text': unicode})
df.dtypes
text               object
Airline            object
name               object
retweet_count     float64
sentiment          object
tweet_location     object
dtype: object

使用 Pandas 读取 UTF-8 数据并将其强制转换为 unicode 的正确方法是什么?

What is the right way of reading and coercing UTF-8 data into unicode with Pandas?

这并不能解决问题:

df = pd.read_csv('1459966468_324.csv', encoding = 'utf8')
df.apply(lambda x: pd.lib.infer_dtype(x.values))

文本文件在这里:https://raw.githubusercontent.com/l1x/nlp/master/1459966468_324.csv

推荐答案

正如其他海报提到的，你可以尝试:

As the other poster mentioned, you might try:

df = pd.read_csv('1459966468_324.csv', encoding='utf8')

但是，当您打印 dtypes 时，这仍然可能让您看到对象".要确认它们是 utf8，请在阅读 CSV 后尝试这一行:

However this could still leave you looking at 'object' when you print the dtypes. To confirm they are utf8, try this line after reading the CSV:

df.apply(lambda x: pd.lib.infer_dtype(x.values))

示例输出:

args            unicode
date         datetime64
host            unicode
kwargs          unicode
operation       unicode

这篇关于如何使用 Pandas 读取 UTF-8 文件?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！