清除格式错误的UTF-8数据 [英] Cleaning malformed UTF-8 data

问题描述

使用Swift，我试图通过 URLSession 获取HTML，而不是将其加载到一个 WKWebView 首先因为我只需要HTML而没有子资源。我遇到了某些页面的问题，这些页面在加载到 WKWebView 时有效，但是当通过 URLSession 加载时（或者甚至一个简单的 NSString（contentsOf：url，编码String.Encoding.utf8.rawValue））UTF-8转换失败。

With Swift, I'm trying to fetch HTML via URLSession rather than by loading it into a WKWebView first as I only need the HTML and none of the subresources. I'm running into a problem with certain pages that work when loaded into WKWebView but when loaded via URLSession (or even a simple NSString(contentsOf: url, encoding String.Encoding.utf8.rawValue)) the UTF-8 conversion fails.

此失败（打印nil）：

This fails (prints "nil"):

print（试试？NSString（contentsOf：URL（字符串：http://www.huffingtonpost.jp/techcrunch-japan/amazon-is-gobbling-whole-foods-for-a-reported-13-7-billion_b_17171132。 html？utm_hp_ref = japan& ir = Japan）!, encoding：String.Encoding.utf8.rawValue））


但是将URL更改为网站的主页，它会成功：

But changing the URL to the site's homepage, it succeeds:

print（试试？NSString（contentsOf：URL（string：http：// www.huffingtonpost.jp）!, encoding：String.Encoding.utf8.rawValue））

如何清理包含格式错误的UTF-8的URL返回的数据？我想删除或替换格式错误的UTF-8中的任何无效序列，以便可以查看其余部分。 WKWebView能够很好地呈现页面（并声称它也是UTF-8内容），您可以通过访问URL看到： http://www.huffingtonpost.jp/techcrunch- japan / amazon-is-gobbling-whole-foods-for-a-reported-13-7-billion_b_17171132.html？utm_hp_ref = japan& ir = Japan

How can I "clean" the data returned by a URL that contains malformed UTF-8? I'd like to either remove or replace any invalid sequences in the malformed UTF-8 so that the rest of it can be viewed. WKWebView is able to render the page just fine (and claims it's UTF-8 content as well), as you can see by visiting the URL: http://www.huffingtonpost.jp/techcrunch-japan/amazon-is-gobbling-whole-foods-for-a-reported-13-7-billion_b_17171132.html?utm_hp_ref=japan&ir=Japan

推荐答案

这是一种从（可能）格式错误的
UTF-8数据创建 String 的方法：

Here is an approach to create a String from (possibly) malformed UTF-8 data:

• 将网站内容读入数据对象。


• 附加 0 字节使其成为C字符串


• 使用字符串（cString :) 用于转换。此初始化程序使用Unicode替换字符（\ u {FFFD}）替换格式错误的UTF-8代码单元序列。


• 可选：删除所有出现的替换字符。

• Read the website contents into a Data object.
• Append a 0 byte to make it a "C string"
• Use String(cString:) for the conversion. This initializer replaces ill-formed UTF-8 code unit sequences with the Unicode replacement character ("\u{FFFD}").
• Optionally: Remove all occurrences of the replacement character.

清理过程示例：

var data = Data(bytes: [65, 66, 200, 67]) // malformed UTF-8

data.append(0)
let s = data.withUnsafeBytes { (p: UnsafePointer<CChar>) in String(cString: p) }
let clean = s.replacingOccurrences(of: "\u{FFFD}", with: "")

print(clean) // ABC

当然这可以定义为自定义init方法：

Of course this can be defined as a custom init method:

extension String {
    init(malformedUTF8 data: Data) {
        var data = data
        data.append(0)
        self = data.withUnsafeBytes { (p: UnsafePointer<CChar>) in
            String(cString: p).replacingOccurrences(of: "\u{FFFD}", with: "")
        }
    }
}

用法：

let data = Data(bytes: [65, 66, 200, 67])
let s = String(malformedUTF8: data) 
print(s) // ABC

清洁可以使用<$ c $更直接完成c> transcode with

The cleaning can be done more "directly" using transcode with

extension String {
    init(malformedUTF8 data: Data) {
        var utf16units = [UInt16]()
        utf16units.reserveCapacity(data.count) // A rough estimate

        _ = transcode(data.makeIterator(), from: UTF8.self, to: UTF16.self,
                      stoppingOnError: false) { code in
            if code != 0xFFFD {
                utf16units.append(code)
            }
        }

        self = String(utf16CodeUnits: utf16units, count: utf16units.count)
    }
}

这基本上是字符串（cString：）
，比较
CString.swift 和
StringCreate.swift 。

另一种选择是使用 UTF8 编解码器 decode（）方法
并忽略错误：

Yet another option is to use the UTF8 codecs decode() method and ignore errors:

extension String {
    init(malformedUTF8 data: Data) {
        var str = ""
        var iterator = data.makeIterator()
        var utf8codec = UTF8()
        var done = false
        while !done {
            switch utf8codec.decode(&iterator) {
            case .emptyInput:
                done = true
            case let .scalarValue(val):
                str.unicodeScalars.append(val)
            case .error:
                break // ignore errors
            }
        }
        self = str
    }
}

这篇关于清除格式错误的UTF-8数据的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！