如何在字符串比较中忽略 UTF-8 字节顺序标记? [英] How do I ignore the UTF-8 Byte Order Marker in String comparisons?

问题描述

我在使用 Visual Studio 2010 比较 C# 4.0 单元测试中的字符串时遇到问题.同样的测试用例在 Visual Studio 2008(使用 C# 3.5)中正常工作.

I'm having a problem comparing strings in a Unit Test in C# 4.0 using Visual Studio 2010. This same test case works properly in Visual Studio 2008 (with C# 3.5).

这是相关的代码片段:

byte[] rawData = GetData();
string data = Encoding.UTF8.GetString(rawData);

Assert.AreEqual("Constant", data, false, CultureInfo.InvariantCulture);

在调试此测试时，data 字符串在肉眼看来包含与文字完全相同的字符串.当我调用 data.ToCharArray() 时，我注意到字符串 data 的第一个字节是值 65279，它是 UTF-8字节顺序标记.我不明白的是为什么 Encoding.UTF8.GetString() 保留这个字节.

While debugging this test, the data string appears to the naked eye to contain exactly the same string as the literal. When I called data.ToCharArray(), I noticed that the first byte of the string data is the value 65279 which is the UTF-8 Byte Order Marker. What I don't understand is why Encoding.UTF8.GetString() keeps this byte around.

如何让 Encoding.UTF8.GetString() 不 将字节顺序标记放入结果字符串中?

How do I get Encoding.UTF8.GetString() to not put the Byte Order Marker in the resulting string?

更新:问题在于从磁盘读取文件的 GetData() 使用 FileStream.readbytes()<从文件中读取数据/代码>.我通过使用 StreamReader 并使用 Encoding.UTF8.GetBytes() 将字符串转换为字节来纠正了这个问题，这是它应该首先做的事情！感谢大家的帮助.

Update: The problem was that GetData(), which reads a file from disk, reads the data from the file using FileStream.readbytes(). I corrected this by using a StreamReader and converting the string to bytes using Encoding.UTF8.GetBytes(), which is what it should've been doing in the first place! Thanks for all the help.

推荐答案

好吧，我认为这是因为原始二进制数据包含 BOM.如果你不想要它，你可以在解码后自己删除 BOM - 但你应该考虑字节数组是否应该考虑 BOM 开始.

Well, I assume it's because the raw binary data includes the BOM. You could always remove the BOM yourself after decoding, if you don't want it - but you should consider whether the byte array should consider the BOM to start with.

或者，您可以使用 StreamReader 来执行解码.这是一个示例，显示了使用 Encoding.GetString 将同一个字节数组转换为两个字符或通过 StreamReader 将一个字符转换为一个字符:

Alternatively, you could use a StreamReader to perform the decoding. Here's an example, showing the same byte array being converted into two characters using Encoding.GetString or one character via a StreamReader:

using System;
using System.IO;
using System.Text;

class Test
{
    static void Main()
    {
        byte[] withBom = { 0xef, 0xbb, 0xbf, 0x41 };
        string viaEncoding = Encoding.UTF8.GetString(withBom);
        Console.WriteLine(viaEncoding.Length);

        string viaStreamReader;
        using (StreamReader reader = new StreamReader
               (new MemoryStream(withBom), Encoding.UTF8))
        {
            viaStreamReader = reader.ReadToEnd();           
        }
        Console.WriteLine(viaStreamReader.Length);
    }
}

这篇关于如何在字符串比较中忽略 UTF-8 字节顺序标记?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！