如何使用 urlopen 获取非 ascii 网址? [英] How to fetch a non-ascii url with urlopen?

问题描述

我需要从包含非 ascii 字符的 URL 中获取数据，但 urllib2.urlopen 拒绝打开资源并引发:

I need to fetch data from a URL with non-ascii characters but urllib2.urlopen refuses to open the resource and raises:

UnicodeEncodeError: 'ascii' codec can't encode character u'\u0131' in position 26: ordinal not in range(128)

我知道 URL 不符合标准，但我没有机会更改它.

I know the URL is not standards compliant but I have no chance to change it.

使用 Python 访问包含非 ascii 字符的 URL 所指向的资源的方法是什么?

What is the way to access a resource pointed by a URL containing non-ascii characters using Python?

换句话说，urlopen可以/如何打开一个像:

edit: In other words, can / how urlopen open a URL like:

http://example.org/Ñöñ-ÅŞÇİİ/

推荐答案

严格来说 URI 不能包含非 ASCII 字符；你有一个IRI.

Strictly speaking URIs can't contain non-ASCII characters; what you have there is an IRI.

要将 IRI 转换为纯 ASCII URI:

To convert an IRI to a plain ASCII URI:

• 地址的主机名部分中的非 ASCII 字符必须使用基于Punycode的编码IDNA算法；

路径中的非 ASCII 字符以及地址的大部分其他部分必须使用 UTF-8 和 %-encoding 进行编码，根据 Ignacio 的回答.

non-ASCII characters in the path, and most of the other parts of the address have to be encoded using UTF-8 and %-encoding, as per Ignacio's answer.

所以:

import re, urlparse

def urlEncodeNonAscii(b):
    return re.sub('[\x80-\xFF]', lambda c: '%%%02x' % ord(c.group(0)), b)

def iriToUri(iri):
    parts= urlparse.urlparse(iri)
    return urlparse.urlunparse(
        part.encode('idna') if parti==1 else urlEncodeNonAscii(part.encode('utf-8'))
        for parti, part in enumerate(parts)
    )

>>> iriToUri(u'http://www.a\u0131b.com/a\u0131b')
'http://www.xn--ab-hpa.com/a%c4%b1b'

(从技术上讲，这在一般情况下仍然不够好，因为 urlparse 不会拆分任何 user:pass@ 前缀或 :port 主机名后缀.只有主机名部分应该是 IDNA 编码的.使用普通的 urllib.quote 和 .encode('idna') 更容易编码您构建 URL 的时间而不是必须将 IRI 分开.)

(Technically this still isn't quite good enough in the general case because urlparse doesn't split away any user:pass@ prefix or :port suffix on the hostname. Only the hostname part should be IDNA encoded. It's easier to encode using normal urllib.quote and .encode('idna') at the time you're constructing a URL than to have to pull an IRI apart.)

这篇关于如何使用 urlopen 获取非 ascii 网址?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！