将以utf-8字符串为内容的unicode转换为str [英] Convert unicode with utf-8 string as content to str

问题描述

我正在使用pyquery来解析页面:

I'm using pyquery to parse a page:

dom = PyQuery('http://zh.wikipedia.org/w/index.php', {'title': 'CSS', 'printable': 'yes', 'variant': 'zh-cn'})
content = dom('#mw-content-text > p').eq(0).text()

但是我在content中得到的是具有utf-8编码内容的unicode字符串:

but what I get in content is a unicode string with utf-8 encoded content:

u'\xe5\xb1\x82\xe5\x8f\xa0\xe6\xa0\xb7\xe5\xbc\x8f\xe8\xa1\xa8...'

如何在不丢失内容的情况下将其转换为str?

how could I convert it to str without lost the content?

明确说明:

我想要conent == '\xe5\xb1\x82\xe5\x8f\xa0\xe6\xa0\xb7\xe5\xbc\x8f\xe8\xa1\xa8'

不是conent == u'\xe5\xb1\x82\xe5\x8f\xa0\xe6\xa0\xb7\xe5\xbc\x8f\xe8\xa1\xa8'

推荐答案

如果您的unicode值具有UTF-8字节，请编码为Latin-1以保留字节":

If you have a unicode value with UTF-8 bytes, encode to Latin-1 to preserve the 'bytes':

content = content.encode('latin1')

因为所有Unicode码点U + 0000至U + 00FF都使用latin-1编码一对一映射；因此，这种编码会将您的数据解释为文字字节.

because the Unicode codepoints U+0000 to U+00FF all map one-on-one with the latin-1 encoding; this encoding thus interprets your data as literal bytes.

以您的示例为例:

>>> content = u'\xe5\xb1\x82\xe5\x8f\xa0\xe6\xa0\xb7\xe5\xbc\x8f\xe8\xa1\xa8'
>>> content.encode('latin1')
'\xe5\xb1\x82\xe5\x8f\xa0\xe6\xa0\xb7\xe5\xbc\x8f\xe8\xa1\xa8'
>>> content.encode('latin1').decode('utf8')
u'\u5c42\u53e0\u6837\u5f0f\u8868'
>>> print content.encode('latin1').decode('utf8')
层叠样式表

PyQuery使用requests或urllib检索HTML，对于requests，使用响应的.text属性.这仅基于Content-Type标头中设置的编码对响应数据进行自动解码，或者如果该信息不可用，则对此使用latin-1(用于文本响应，但HTML是文本响应).您可以通过传递一个encoding参数来覆盖它:

PyQuery uses either requests or urllib to retrieve the HTML, and in the case of requests, uses the .text attribute of the response. This auto-decodes the response data based on the encoding set in a Content-Type header alone, or if that information is not available, uses latin-1 for this (for text responses, but HTML is a text response). You can override this by passing in an encoding argument:

dom = PyQuery('http://zh.wikipedia.org/w/index.php', encoding='utf8',
              {'title': 'CSS', 'printable': 'yes', 'variant': 'zh-cn'})

这时您根本不需要重新编码.

at which point you'd not have to re-encode at all.

这篇关于将以utf-8字符串为内容的unicode转换为str的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！