协议缓冲区和UTF-8 [英] Protocol buffers and UTF-8

问题描述

编码方案/多操作系统和端序的历史导致了编码所有形式的字符串数据（即所有字母表）的混乱;因为协议缓冲区在它的字符串类型只处理ASCII或UTF-8，我不能看到任何接受C ++ wstring的多态重载。问题是如何预期得到一个UTF-16字符串到协议缓冲区？

可能我需要将数据作为wstring保存在我的应用程序代码中，然后执行UTF-8转换，然后再将其填入（或从中提取）消息。什么是最简单的 - Windows / Linux可移植的方式来做这个（从一个良好支持的库的单个函数调用将使我的一天）？

数据将来自各种Web服务器（Linux和Windows），最终将以SQL Server（以及可能的其他端点）结尾。

- 编辑1 -

Mark Wilkins的建议似乎符合帐单，也许有人在图书馆有经验的人可以张贴

- 编辑2 -

$ b代码片段 - 从wstring到UTF-8 -
$ b

sth的建议更多。我会进一步调查boost序列化。

解决方案

这可能是过度，但 ICU库会做你需要的一切，你可以在Windows和Linux上使用它们。

但是，如果你只想要转换，那么在Windows下，对 MultiByteToWideChar 和 WideCharToMultiByte 的简单调用可以进行转换UTF-8和

UTF-16之间。例如：

  // utf-8至utf-16 
 MultiByteToWideChar（CP_UTF8，0，myUtf8String，-1 ，
 myUtf16Buf，lengthOfUtf16Buf）; 
  

对于Linux，libidn可以做你需要的。它可以在UTF-8和UCS之间转换，我认为在某个级别等同于UTF-32。例如：

  // utf-8 to UCS 
 ucsStr = stringprep_utf8_to_ucs4（asdf，4，&项目）;但是，在Linux中，我认为你可能最好只是使用UTF-8工作。除非你有一个现有的库为UTF-16，我不知道有一个令人信服的理由在Linux中使用它。
 
The history of Encoding Schemes / multiple Operating Systems and Endian-nes have led to a mess in terms of encoding all forms of string data (--i.e., all alphabets); for this reason protocol buffers only deals with ASCII or UTF-8 in its string types, and I can't see any polymorphic overloads that accept the C++ wstring. The question then is how is one expected to get a UTF-16 string into a protocol buffer ? 

Presumably I need to keep the data as a wstring in my application code and then perform a UTF-8 conversion before I stuff it into (or extract from) the message. What is the simplest - Windows/Linux portable way to do this (A single function call from a well-supported library would make my day) ? 

Data will originate from various web-servers (Linux and windows) and will eventually ends up in SQL Server (and possibly other end points).

-- edit 1--

Mark Wilkins suggestion seems to fit the bill, perhaps someone who has experience with the library can post a code snippet -- from wstring to UTF-8 -- so that I can gauge how easy it will be.

-- edit 2 --

sth's suggestion even more so. I will investigate boost serialization further.
解决方案
It may be overkill, but the ICU libraries will do everything you need and you can use them on both Windows and Linux.

However, if you are only wanting conversion, then under Windows, a simple call to MultiByteToWideChar and WideCharToMultiByte can do the conversion between UTF-8 and 

UTF-16.  For example:
// utf-8 to utf-16
MultiByteToWideChar( CP_UTF8, 0, myUtf8String, -1,
                     myUtf16Buf, lengthOfUtf16Buf );
With Linux, libidn might do what you need.  It can convert between UTF-8 and UCS, which I think is equivalent to UTF-32 at some level.  For example:
// utf-8 to UCS
ucsStr = stringprep_utf8_to_ucs4( "asdf", 4, &items );
However, in Linux I think you might be best simply working with UTF-8.  Unless you have an existing library for UTF-16, I am not sure there is a compelling reason to use it in Linux.

                        
这篇关于协议缓冲区和UTF-8的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！