相同的单词但不同的 Unicode 字符 [英] Same word but different unicode characters

问题描述

我在 Windows 上使用 Python 构建了一个关于越南餐馆的问答应用程序.要写越南语字符，我需要使用 Unicode.
首先，我从使用 HTML charset=utf-8 的 TripAdvisor 网站克隆数据并构建我的 Mongo 数据库.TripAdvisor 中名为đà nẵng"的城市有一个代码:

<预><代码>>>>print repr("đà nẵng") # 来自 tripadvisor 网站>>>'\xc4\x91a\xcc\x80 n\xc4\x83\xcc\x83ng'

然而，当我从 Firefox 的地址栏查询时，城市đà nẵng"有不同的代码:

<预><代码>>>>print repr("đà nẵng") # Firefox 的地址栏>>>'\xc4\x91\xc3\xa0 n\xe1\xba\xb5ng'

这就是我在我的数据库中找不到那个城市的原因.我尝试在记事本++上写这个城市名称并得到与使用Firefox地址栏相同的结果

<预><代码>>>>print repr("đà nẵng") # notepad++ using 'Encoding UTF-8'>>>'\xc4\x91\xc3\xa0 n\xe1\xba\xb5ng'

有没有办法在两种类型的代码之间进行转换?
或者在这种情况下有什么方法可以将城市名称đà nẵng"与不同的代码相匹配?.

解决方案

您遇到的问题是，unicode 允许以多种方式组合同一个符号.Python 模块 unicodedata 提供了一个函数 normalize，允许您将 unicode 表示转换为固定的 表单(例如NFC)

from unicodedata import normalizeS1 = b'\xc4\x83\xcc\x83'.decode('UTF-8')S2 = b'\xe1\xba\xb5'.decode('UTF-8')打印(标准化('NFC'，S1).编码('UTF-8'))打印(标准化('NFC'，S2).编码('UTF-8'))

在您的示例中，tripadvisor 以 NFD 形式显示，而记事本使用的是 NFC.

I built a question answering application in terms of restaurants in Vietnam using Python on Windows. To write Vietnamese characters I need to use Unicode.
First, I clone data from TripAdvisor website which used HTML charset=utf-8 and build my Mongo database. A city named "đà nẵng" in TripAdvisor has a code:

>>> print repr("đà nẵng")     # from tripadvisor website 
>>> '\xc4\x91a\xcc\x80 n\xc4\x83\xcc\x83ng'

However, when I query from Firefox's address bar, the city "đà nẵng" has a different code:

>>> print repr("đà nẵng")   # Firefox's address bar
>>> '\xc4\x91\xc3\xa0 n\xe1\xba\xb5ng'

That is a reason why I can not find that city in my database. I try to write this city name on notepad++ and got the same result as using Firefox's address bar

>>> print repr("đà nẵng")   # notepad++ using 'Encoding UTF-8'
>>> '\xc4\x91\xc3\xa0 n\xe1\xba\xb5ng'

Is there any way to convert between two types of code?
Or is there any way to match the city name "đà nẵng" with different codes in this case?.

解决方案

The problem you encounter is, that unicode allows multiple ways to compose the same symbol. The Python module unicodedata provides a function normalize that allows you to convert unicode representations to a fixed form (e.g. NFC)

from unicodedata import normalize

S1 = b'\xc4\x83\xcc\x83'.decode('UTF-8')
S2 = b'\xe1\xba\xb5'.decode('UTF-8')

print(normalize('NFC', S1).encode('UTF-8'))
print(normalize('NFC', S2).encode('UTF-8'))

In your example tripadvisor displayed in NFD form, while notepad used NFC.

这篇关于相同的单词但不同的 Unicode 字符的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！