const std :: wstring如何编码以及如何更改为UTF-16 [英] How is const std::wstring encoded and how to change to UTF-16
问题描述
我创建了这个最低限度的C ++示例代码段,以在定义字符串时比较 std :: string
和 std :: wstring
中的字节(以十六进制表示)可以使用两种类型的德语非ASCII字符.
I created this minimum working C++ example snippet to compare bytes (by their hex representation) in a std::string
and a std::wstring
when defining a string with german non-ASCII characters in either type.
#include <iostream>
#include <iomanip>
#include <string>
int main(int, char**) {
std::wstring wstr = L"äöüß";
std::string str = "äöüß";
for ( unsigned char c : str ) {
std::cout << std::setw(2) << std::setfill('0') << std::hex << static_cast<unsigned short>(c) << ' ';
}
std::cout << std::endl;
for ( wchar_t c : wstr ) {
std::cout << std::setw(4) << std::setfill('0') << std::hex << static_cast<unsigned short>(c) << ' ';
}
std::cout << std::endl;
return 0;
}
此代码段的输出为
c3 a4 c3 b6 c3 bc c3 9f
00c3 00a4 00c3 00b6 00c3 00bc 00c3 0178
我在运行 Windows 10 64位Pro 的PC上运行此程序,并使用构建系统在版本16.8.1中的 MSVC 2019社区版上进行了编译cmake 和以下 CMakeLists.txt
I ran this on a PC running itself Windows 10 64-bit Pro, compiling with MSVC 2019 Community Edition in Version 16.8.1, using build system cmake with following CMakeLists.txt
cmake_minimum_required(VERSION 3.0.0)
project(wstring VERSION 0.1.0)
set(CMAKE_CXX_STANDARD 17)
include(CTest)
enable_testing()
add_executable(wstring main.cpp)
set(CPACK_PROJECT_NAME ${PROJECT_NAME})
set(CPACK_PROJECT_VERSION ${PROJECT_VERSION})
include(CPack)
我读到, std :: string
是基于 char
类型的,它是一个字节.我看到我的代码段的输出表明 str
( std :: string
变量)是 UTF-8 编码的.我读到,Microsoft编译器使用2字节的 wchar_t
s组成 std :: wstring
s(而不是4字节的 wchar_t
s例如GNU gcc),因此期望 wstr
( std :: wstring
变量)经过(任何类型) UTF-16 编码.但我无法弄清楚为什么ß"字样会(拉丁字母s)被编码为 0x00c30178
,而我原来期望的是 0x00df
.可能有人请告诉我:
I read, that std::string
s are based on char
type which is a single byte. I see that the output of my snippet indicates that str
(the std::string
variable) is UTF-8 encoded. I read up, that Microsoft compilers use wchar_t
s with 2 bytes to make up std::wstring
s (instead of 4 byte wchar_t
s by e.g. GNU gcc) and therefore would expect wstr
(the std::wstring
variable) to be (any kind of) UTF-16 encoded. But I cannot figure out why the "ß" (latin sharp s) is encoded as 0x00c30178
I had expected 0x00df
instead. May somebody please tell me:
- 为什么会这样?
- 如何最终得到UTF-16编码的
std :: wstring
s(Big Endian可以,我不介意BOM)?我可能需要以某种方式告诉编译器吗? - 这是哪种编码?
- Why this is happening?
- How may I end up with UTF-16 encoded
std::wstring
s (Big Endian would be fine, I do not mind a BOM)? Do I probably need to tell the compiler somehow? - What kind of encoding is this?
更改了标题,因为它不适合问题(实际上UTF-8和UTF-16是不同的编码,所以我已经是新答案了……)
changed title, as it did not fit the questions properly (and actually UTF-8 and UTF-16 are different encodings so the I my self new the answer already...)
忘了提及:我使用提到的编译器的 amd64
目标
forgot to mention: I use the amd64
target of the mentioned compiler
如果添加dxiv注释中指出的/utf-8
标志(请参阅已链接SO-Post ),我得到了所需的输出
if adding the /utf-8
flag as pointed out in the comments by dxiv (see his linked SO-Post), I get the desired output
c3 a4 c3 b6 c3 bc c3 9f
00e4 00f6 00fc 00df
对我来说看起来像UTF-16-BE(无BOM).由于我对cmake命令的正确顺序有疑问,这是我当前的 CmakeLists.txt
文件.重要的是将 add_compile_options
命令放在 add_executable
命令之前(为方便起见,我添加了通知)
which looks like UTF-16-BE (no BOM) for me. As I had issues with the correct order of cmake commands this is my current CmakeLists.txt
file. It is important to put the add_compile_options
command before the add_executable
command (I added the Notice for convenience)
cmake_minimum_required(VERSION 3.0.0)
project(enctest VERSION 0.1.0)
set(CMAKE_CXX_STANDARD 17)
include(CTest)
enable_testing()
if (MSVC)
message(NOTICE "compiling with MSVC")
add_compile_options(/utf-8)
endif()
add_executable(enctest main.cpp)
set(CPACK_PROJECT_NAME ${PROJECT_NAME})
set(CPACK_PROJECT_VERSION ${PROJECT_VERSION})
include(CPack)
我发现 if-endif
的方式比生成器语法更易读,但是编写了 add_compile_options("$< $< CXX_COMPILER_ID:MSVC> ::/utf-8>''')
代替也可以.
I find the if-endif
way more readable, than the generator-syntax one, but writing add_compile_options("$<$<CXX_COMPILER_ID:MSVC>:/utf-8>")
instead would work as well.
注意:对于Qt项目, .pro
文件有一个不错的选择(请参见
Note: For Qt-Projects there is a nice switch for the .pro
file (see this Qt-Form post)
win32 {
QMAKE_CXXFLAGS += /utf-8
}
我的问题的第一部分仍然是开放的:对于ß", 0x00c30178
是什么编码?(拉丁语尖锐的s)?
Still the first part of my question is open: What encoding is 0x00c30178
for "ß" (latin sharp s)?
推荐答案
正如注释中所阐明的,源 .cpp
文件是UTF-8编码的.没有BOM,也没有显式的/source-charset:utf-8
开关,Visual C ++编译器默认假定源文件保存在活动代码页编码中.从设置源字符集文档:
As clarified in the comments, the source .cpp
file is UTF-8 encoded. Without a BOM, and without an explicit /source-charset:utf-8
switch, the Visual C++ compiler defaults to assuming the source file is saved in the active codepage encoding. From the Set Source Character Set documentation:
默认情况下,Visual Studio将检测一个字节顺序标记,以确定源文件是否为编码的Unicode格式,例如UTF-16或UTF-8.如果未找到字节序标记,则除非使用/source-charset选项指定字符集名称或代码页,否则将假定源文件是使用当前用户代码页编码的.
By default, Visual Studio detects a byte-order mark to determine if the source file is in an encoded Unicode format, for example, UTF-16 or UTF-8. If no byte-order mark is found, it assumes the source file is encoded using the current user code page, unless you specify a character set name or code page by using the /source-charset option.
äöüß
的UTF-8编码为 C3 A4 C3 B6 C3 BC C3 9F
,因此该行:
The UTF-8 encoding of äöüß
is C3 A4 C3 B6 C3 BC C3 9F
, and therefore the line:
std::wstring wstr = L"äöüß";
被编译器视为:
std::wstring wstr = L"\xC3\xA4\xC3\xB6\xC3\xBC\xC3\x9F"`;
假设活动代码页是通常的 Windows-1252 ,(扩展)字符映射为:
Assuming the active codepage to be the usual Windows-1252, the (extended) characters map as:
win-1252 char unicode
\xC3 Ã U+00C3
\xA4 ¤ U+00A4
\xB6 ¶ U+00B6
\xBC ¼ U+00BC
\x9F Ÿ U+0178
因此, L"\ xC3 \ xA4 \ xC3 \ xB6 \ xC3 \ xBC \ xC3 \ x9F""
被翻译为:
std::wstring wstr = L"\u00C3\u00A4\u00C3\u00B6\u00C3\u00BC\u00C3\u0178"`;
为避免这种(错误)翻译,需要通过传递显式的/source-charset:utf-8
(或/utf-8
)编译器开关.对于基于CMake的项目,可以使用 add_compile_options
完成,如
To avoid such (mis)translation, Visual C++ needs to be told that the source file is encoded as UTF-8 by passing an explicit /source-charset:utf-8
(or /utf-8
) compiler switch. For CMake based projects, this can be done using add_compile_options
as shown at Possible to force CMake/MSVC to use UTF-8 encoding for source files without a BOM? C4819.
这篇关于const std :: wstring如何编码以及如何更改为UTF-16的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!