删除无效/不完整的多字节字符 [英] Removing invalid/incomplete multibyte characters

问题描述

在用户输入端使用以下代码时，我遇到了一些问题:

I'm having some issues using the following code on user input:

htmlentities($string, ENT_COMPAT, 'UTF-8');

当检测到无效的多字节字符时，PHP会发出通知:

When an invalid multibyte character is detected PHP throws a notice:

PHP警告:htmlentities():第123行的/path/to/file.php中的参数中的无效多字节序列

PHP Warning: htmlentities(): Invalid multibyte sequence in argument in /path/to/file.php on line 123

我的第一个想法是抑制该错误，但这是缓慢且较差的做法: http://derickrethans.nl/five-reasons-why-the- shutop-operator-should-be-avoided.html

My first thought was to supress the error, but this is slow and poor practice: http://derickrethans.nl/five-reasons-why-the-shutop-operator-should-be-avoided.html

我的第二个想法是使用ENT_IGNORE标志，但是即使PHP手册也建议不要使用此标志:

My second thought was to use the ENT_IGNORE flag, but even the PHP manual suggests not to use this:

安静地丢弃无效的代码单元序列，而不是返回空字符串.不建议使用此标志，因为它可能会带来安全隐患.

进一步的原因导致我进入以下代码段:

A bit of further reason led me to the following piece of code:

    // detect encoding
$encoding =  mb_detect_encoding($query);
if($encoding != 'UTF-8') {
    $query = mb_convert_encoding($query, 'UTF-8', $encoding);
} else {
    // strip out invalid utf8 sequences
    $query = iconv('UTF-8', 'UTF-8//IGNORE', $query);
}

不幸的是，iconv 也在删除/忽略无效字符时抛出E_NOTICE:

Unfortunately iconv also throws an E_NOTICE when it removes/ignores invalid characters:

如果将字符串//TRANSLIT附加到out_charset，则会激活音译.这意味着当一个字符不能在目标字符集中表示时，可以通过一个或几个外观相似的字符来近似.如果附加字符串//IGNORE，则无法在目标字符集中表示的字符将被静默丢弃.否则，将从第一个非法字符中截取str并生成一个E_NOTICE.

If you append the string //TRANSLIT to out_charset transliteration is activated. This means that when a character can't be represented in the target charset, it can be approximated through one or several similarly looking characters. If you append the string //IGNORE, characters that cannot be represented in the target charset are silently discarded. Otherwise, str is cut from the first illegal character and an E_NOTICE is generated.

因此，我基本上没有其他选择.我宁愿使用久经考验的库来处理此类问题，而不是尝试使用我见过的一些基于正则表达式的解决方案来进行尝试.

So I'm basically out of options here. I'd rather use a tried and tested library to handle this kind of stuff than attempting it with a few of the regular expression based solutions I've seen floated around.

这使我想到了最后一个问题: 如何有效，安全地删除无效的多字节字符，而不会发出通知/警告/错误?

So that leads me to my final question: How can I remove invalid multibyte characters, efficiently, securely, without notices/warnings/errors?

推荐答案

如何有效，安全地删除无效的多字节字符，而不会发出通知/警告/错误?

How can I remove invalid multibyte characters, efficiently, securely, without notices/warnings/errors?

好吧，正如您已经在问题中概述的那样(或至少已链接)，则不能删除无效字节序列.

Well, as you already have outlined in your question on your own (or at least linked), deleting the invalid byte sequence(s) is not an option.

相反，应该用替换字符U + FFFD替换它.从PHP 5.4.0开始，您可以对 htmlentities 使用ENT_SUBSTITUTE标志.如果您不想拒绝该字符串，那可能是最安全的.

Instead it should be probably replaced with the replacement character U+FFFD. As of PHP 5.4.0 you can make use of the ENT_SUBSTITUTE flag for htmlentities. That's probably most safe if you don't want to reject the string.

iconv在最近的PHP版本中始终会警告您，即使没有删除整个字符串也是如此.因此，对于您来说，它似乎不是一个很好的选择.

iconv will always give you warning in recent PHP versions if not even deleting the whole string. So it does not look like a good alternative for you.

这篇关于删除无效/不完整的多字节字符的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！