检查无效的 UTF8 [英] Check for invalid UTF8

问题描述

我正在从 UTF8 格式转换为十六进制的实际值.但是，我需要捕获一些无效的字节序列.有没有一种快速的方法来检查一个字符是否不属于 C++ 中的 UTF8?

I am converting from UTF8 format to actual value in hex. However there are some invalid sequences of bytes that I need to catch. Is there a quick way to check if a character doesn't belong in UTF8 in C++?

推荐答案

遵循 Unicode 标准章节中的表格3.(我使用了章节(p103)的 Unicode 5.1.0 版本；它在 Unicode 6.0.0 版本的 p94 上是表 3-7，在 Unicode 6.3 版本中是在 p95 上——它在 p125 上Unicode 8.0.0 版本.)

Follow the tables in the Unicode standard, chapter 3. (I used the Unicode 5.1.0 version of the chapter (p103); it was Table 3-7 on p94 of the Unicode 6.0.0 version, and was on p95 in the Unicode 6.3 version — and it is on p125 of the Unicode 8.0.0 version.)

字节 0xC0、0xC1 和 0xF5..0xFF 不能出现在有效的 UTF-8 中.记录有效序列；其他都无效.

Bytes 0xC0, 0xC1, and 0xF5..0xFF cannot appear in valid UTF-8. The valid sequences are documented; all others are invalid.

Code Points        First Byte Second Byte Third Byte Fourth Byte
U+0000..U+007F     00..7F
U+0080..U+07FF     C2..DF     80..BF
U+0800..U+0FFF     E0         A0..BF      80..BF
U+1000..U+CFFF     E1..EC     80..BF      80..BF
U+D000..U+D7FF     ED         80..9F      80..BF
U+E000..U+FFFF     EE..EF     80..BF      80..BF
U+10000..U+3FFFF   F0         90..BF      80..BF     80..BF
U+40000..U+FFFFF   F1..F3     80..BF      80..BF     80..BF
U+100000..U+10FFFF F4         80..8F      80..BF     80..BF

请注意，对于第一个字节的某些值范围，不规则位于第二个字节中.需要时，第三个和第四个字节是一致的.请注意，并非标识为有效范围内的每个代码点都已分配(有些是明确的非字符")，因此还需要更多验证.

Note that the irregularities are in the second byte for certain ranges of values of the first byte. The third and fourth bytes, when needed, are consistent. Note that not every code point within the ranges identified as valid has been allocated (and some are explicitly 'non-characters'), so there is more validation needed still.

代码点 U+D800..U+DBFF 用于 UTF-16 高代理，U+DC00..U+DFFF 用于 UTF-16 低代理；那些不能出现在有效的 UTF-8 中(您直接在 UTF-8 中对 BMP(基本多语言平面)之外的值进行编码)，这就是该范围被标记为无效的原因.

The code points U+D800..U+DBFF are for UTF-16 high surrogates and U+DC00..U+DFFF are for UTF-16 low surrogates; those cannot appear in valid UTF-8 (you encode the values outside the BMP — Basic Multilingual Plane — directly in UTF-8), which is why that range is marked invalid.

其他排除范围(初始字节 C0 或 C1，或初始字节 E0 后跟 80..9F，或初始字节 F0 后跟 80..8F)是非最小编码.例如，C0 80 将编码 U+0000，但它由 00 编码，并且 UTF-8 定义非最小编码 C0 80 是无效的.并且最大的 Unicode 码位是 U+10FFFF；从 F4 90 开始的 UTF-8 编码会生成超出范围的值.

Other excluded ranges (initial byte C0 or C1, or initial byte E0 followed by 80..9F, or initial byte F0 followed by 80..8F) are non-minimal encodings. For example, C0 80 would encode U+0000, but that's encoded by 00, and UTF-8 defines that the non-minimal encoding C0 80 is invalid. And the maximum Unicode code point is U+10FFFF; UTF-8 encodings starting from F4 90 upwards generate values that are out of range.

这篇关于检查无效的 UTF8的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！