使用 numpy.genfromtxt 在 Python 3 中加载 UTF-8 文件 [英] Loading UTF-8 file in Python 3 using numpy.genfromtxt

问题描述

我有一个从 WHO 网站下载的 CSV 文件 (http://apps.who.int/gho/data/view.main.52160 ，下载，CSV 格式的多用途表").我尝试将文件加载到一个 numpy 数组中.这是我的代码:

I have a CSV file that I downloaded from WHO site (http://apps.who.int/gho/data/view.main.52160 , Downloads, "multipurpose table in CSV format"). I try to load the file into a numpy array. Here's my code:

import numpy
#U75 - unicode string of max. length 75
world_alcohol = numpy.genfromtxt("xmart.csv", dtype="U75", skip_header=2, delimiter=",")
print(world_alcohol)

我明白了

UnicodeDecodeError: 'ascii' 编解码器无法解码字节 0xc3 的位置2:序数不在范围内(128).

UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 2: ordinal not in range(128).

我猜 numpy 在读取字符串Côte d'Ivoire"时有问题.该文件已正确编码为 UTF-8(根据我的文本编辑器).我使用的是 Python 3.4.3 和 numpy 1.9.2.

I guess that numpy has a problem reading the string "Côte d'Ivoire". The file is properly encoded UTF-8 (according to my text editor). I am using Python 3.4.3 and numpy 1.9.2.

我做错了什么?如何将文件读入 numpy?

What am I doing wrong? How can I read the file into numpy?

推荐答案

请注意原始 2015 年日期.从那时起 genfromtxt 获得了一个 encoding 参数.

Note the original 2015 date. Since then genfromtxt has gotten an encoding parameter.

在 Python3 中我可以做到:

In Python3 I can do:

In [224]: txt = "Côte d'Ivoire"
In [225]: x = np.zeros((2,),dtype='U20')
In [226]: x[0] = txt
In [227]: x
Out[227]: 
array(["Côte d'Ivoire", ''],   dtype='<U20')

这意味着我可能可以打开一个UTF-8"文件(常规，非字节模式)和 readlines，并将它们分配给像 x 这样的数组元素.

Which means I probably could open a 'UTF-8' file (regular, not byte mode), and readlines, and assign them to elements of an array like x.

但是 genfromtxt 坚持使用无法处理更大的 UTF-8 集(7 字节 v 8)的字节字符串 (ascii) 进行操作.所以我需要在某个时候应用 decode 来获得一个 U 数组.

But genfromtxt insists on operating with byte strings (ascii) which can't handle the larger UTF-8 set (7 bytes v 8). So I need to apply decode at some point to get an U array.

我可以使用 genfromtxt 将它加载到一个 'S' 数组中:

I can load it into a 'S' array with genfromtxt:

In [258]: txt="Côte d'Ivoire"
In [259]: a=np.genfromtxt([txt.encode()],delimiter=',',dtype='S20')
In [260]: a
Out[260]: 
array(b"Cxc3xb4te d'Ivoire",  dtype='|S20')

并将 decode 应用于单个元素:

and apply decode to individual elements:

In [261]: print(a.item().decode())
Côte d'Ivoire

In [325]: print _
Côte d'Ivoire

或者使用 np.char.decode 将其应用于数组的每个元素:

Or use np.char.decode to apply it to each element of an array:

In [263]: np.char.decode(a)
Out[263]: 
array("Côte d'Ivoire", dtype='<U13')
In [264]: print(_)
Côte d'Ivoire

genfromtxt 让我指定 converters:

In [297]: np.genfromtxt([txt.encode()],delimiter=',',dtype='U20',
    converters={0:lambda x: x.decode()})
Out[297]: 
array("Côte d'Ivoire", dtype='<U20')

如果 csv 混合了字符串和数字，这种 converters 方法将比 np.char.decode 更容易使用.只需为每个字符串列指定转换器.

If the csv has a mix of strings and numbers, this converters approach will be easier to use than the np.char.decode. Just specify the converter for each string column.

(请参阅我之前对 Python2 尝试所做的编辑).

(See my earlier edits for Python2 tries).

这篇关于使用 numpy.genfromtxt 在 Python 3 中加载 UTF-8 文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！