在Perl中解码UTF-8 JSON的问题 [英] Problem with decoding UTF-8 JSON in perl

问题描述

使用JSON库处理时，UTF-8字符被破坏(也许这类似于

UTF-8 characters are destroyed when processed with the JSON library (maybe this is similar to Problem with decoding unicode JSON in perl, however setting binmode only creates another problem).

我已将问题简化为以下示例:

I have reduced the problem down to the following example:

(hlovdal) localhost:/tmp/my_test>cat my_test.pl
#!/usr/bin/perl -w
use strict;
use warnings;
use JSON;
use File::Slurp;
use Getopt::Long;
use Encode;

my $set_binmode = 0;
GetOptions("set-binmode" => \$set_binmode);

if ($set_binmode) {
        binmode(STDIN,  ":encoding(UTF-8)");
        binmode(STDOUT, ":encoding(UTF-8)");
        binmode(STDERR, ":encoding(UTF-8)");
}

sub check {
        my $text = shift;
        return "is_utf8(): " . (Encode::is_utf8($text) ? "1" : "0") . ", is_utf8(1): " . (Encode::is_utf8($text, 1) ? "1" : "0"). ". ";
}

my $my_test = "hei på deg";
my $json_text = read_file('my_test.json');
my $hash_ref = JSON->new->utf8->decode($json_text);

print check($my_test), "\$my_test = $my_test\n";
print check($json_text), "\$json_text = $json_text";
print check($$hash_ref{'my_test'}), "\$\$hash_ref{'my_test'} = " . $$hash_ref{'my_test'} .  "\n";

(hlovdal) localhost:/tmp/my_test>

在运行测试时，由于某种原因，该文本被误写为iso-8859-1.设置binmode排序方式可以解决该问题，但随后会导致其他字符串进行双重编码.

When running testing the text is for some reason crippeled into iso-8859-1. Setting binmode sort of solves it but then causes double encoding of other strings.

(hlovdal) localhost:/tmp/my_test>cat my_test.json 
{ "my_test" : "hei på deg" }
(hlovdal) localhost:/tmp/my_test>file my_test.json 
my_test.json: UTF-8 Unicode text
(hlovdal) localhost:/tmp/my_test>hexdump -c my_test.json 
0000000   {       "   m   y   _   t   e   s   t   "       :       "   h
0000010   e   i       p 303 245       d   e   g   "       }  \n        
000001e
(hlovdal) localhost:/tmp/my_test>
(hlovdal) localhost:/tmp/my_test>perl my_test.pl
is_utf8(): 0, is_utf8(1): 0. $my_test = hei på deg
is_utf8(): 0, is_utf8(1): 0. $json_text = { "my_test" : "hei på deg" }
is_utf8(): 1, is_utf8(1): 1. $$hash_ref{'my_test'} = hei p� deg
(hlovdal) localhost:/tmp/my_test>perl my_test.pl --set-binmode
is_utf8(): 0, is_utf8(1): 0. $my_test = hei på deg
is_utf8(): 0, is_utf8(1): 0. $json_text = { "my_test" : "hei på deg" }
is_utf8(): 1, is_utf8(1): 1. $$hash_ref{'my_test'} = hei på deg
(hlovdal) localhost:/tmp/my_test>

这是什么原因以及如何解决?

这是在新安装且最新的Fedora 15系统上.

This is on a newly installed and up to date Fedora 15 system.

(hlovdal) localhost:/tmp/my_test>perl --version | grep version
This is perl 5, version 12, subversion 4 (v5.12.4) built for x86_64-linux-thread-multi
(hlovdal) localhost:/tmp/my_test>rpm -q perl-JSON
perl-JSON-2.51-1.fc15.noarch
(hlovdal) localhost:/tmp/my_test>locale
LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=
(hlovdal) localhost:/tmp/my_test>

---

更新:添加use utf8不能解决问题，仍然无法正确处理字符(尽管与之前略有不同):

---

Update: Adding use utf8 does not solve it, characters are still not processed right (although slightly different from before):

(hlovdal) localhost:/tmp/my_test>perl  my_test.pl
is_utf8(): 1, is_utf8(1): 1. $my_test = hei p� deg
is_utf8(): 0, is_utf8(1): 0. $json_text = { "my_test" : "hei på deg" }
is_utf8(): 1, is_utf8(1): 1. $$hash_ref{'my_test'} = hei p� deg
(hlovdal) localhost:/tmp/my_test>perl  my_test.pl --set-binmode
is_utf8(): 1, is_utf8(1): 1. $my_test = hei på deg
is_utf8(): 0, is_utf8(1): 0. $json_text = { "my_test" : "hei på deg" }
is_utf8(): 1, is_utf8(1): 1. $$hash_ref{'my_test'} = hei på deg
(hlovdal) localhost:/tmp/my_test>

如 perlunifaq

我可以在Perl源代码中使用Unicode吗?

Can I use Unicode in my Perl sources?

是的，可以！如果您的来源是 以UTF-8编码，您可以指出 使用utf8编译指示.

Yes, you can! If your sources are UTF-8 encoded, you can indicate that with the use utf8 pragma.

use utf8;

这对您没有任何帮助 输入或输出.它只是 影响您的信息来源的方式 读.您可以在字符串中使用Unicode 文字，在标识符中(但它们 仍然必须是文字字符" 根据\ w)，甚至是自定义 定界符.

This doesn't do anything to your input, or to your output. It only influences the way your sources are read. You can use Unicode in string literals, in identifiers (but they still have to be "word characters" according to \w ), and even in custom delimiters.

推荐答案

问题的核心是JSON对八位字节数组而不是字符串的期望(在

The core of the problem was JSON's expectation of an octet array instead of character string (solved in this question). However I was also missing several things related to unicode, like "use utf8". Here is the diff required to make the code in the example work fully:

--- my_test.pl.orig 2011-08-03 15:44:44.217868886 +0200
+++ my_test.pl  2011-08-03 15:55:30.152379269 +0200
@@ -1,19 +1,14 @@
-#!/usr/bin/perl -w
+#!/usr/bin/perl -CSAD
 use strict;
 use warnings;
 use JSON;
 use File::Slurp;
 use Getopt::Long;
 use Encode;
-
-my $set_binmode = 0;
-GetOptions("set-binmode" => \$set_binmode);
-
-if ($set_binmode) {
-        binmode(STDIN,  ":encoding(UTF-8)");
-        binmode(STDOUT, ":encoding(UTF-8)");
-        binmode(STDERR, ":encoding(UTF-8)");
-}
+use utf8;
+use warnings qw< FATAL utf8 >;
+use open qw( :encoding(UTF-8) :std );
+use feature qw< unicode_strings >;

 sub check {
         my $text = shift;
@@ -21,8 +16,9 @@
 }

 my $my_test = "hei på deg";
-my $json_text = read_file('my_test.json');
-my $hash_ref = JSON->new->utf8->decode($json_text);
+my $json_text = read_file('my_test.json', binmode => ':encoding(UTF-8)');
+my $json_bytes = encode('UTF-8', $json_text);
+my $hash_ref = JSON->new->utf8->decode($json_bytes);

 print check($my_test), "\$my_test = $my_test\n";
 print check($json_text), "\$json_text = $json_text";

这篇关于在Perl中解码UTF-8 JSON的问题的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！