Facebook JSON 编码错误 [英] Facebook JSON badly encoded

问题描述

我下载了我的 Facebook Messenger 数据(在您的 Facebook 帐户中，转到设置，然后转到您的 Facebook 信息，然后下载您的信息，然后创建一个至少选中消息框的文件)来做一些很酷的统计

但是编码有一个小问题.我不确定，但看起来 Facebook 对这些数据使用了错误的编码.当我用文本编辑器打开它时，我看到如下内容:Rados\u00c5\u0082aw.当我尝试用 python (UTF-8) 打开它时，我得到 RadosÅ\x82aw.但是我应该得到:Radosław.

我的python脚本:

text = open(os.path.join(subdir, file), encoding='utf-8')对话.附加(json.load(文本))

我尝试了一些最常见的编码.示例数据是:

<代码>{"sender_name": "雷达\u00c5\u0082aw",时间戳":1524558089，"content": "拒绝 trzeba ostatnie treningi zrobi\u00c4\u0087 xD",类型":通用"}

解决方案

我确实可以确认 Facebook 下载数据编码不正确；Mojibake.原始数据采用 UTF-8 编码，但被解码为拉丁文 -1.我会确保提交错误报告.

同时，您可以通过两种方式修复损坏:

1. 将数据解码为 JSON，然后将任何字符串重新编码为 Latin-1，再次解码为 UTF-8:

   <预><代码>>>>导入json>>>数据 = r'"雷达\u00c5\u0082aw"'>>>json.loads(data).encode('latin1').decode('utf8')'拉多斯瓦夫'

2. 以二进制形式加载数据，将所有\u00hh序列替换为最后两个十六进制数字表示的字节，解码为UTF-8，然后解码为JSON:

   导入重新从 functools 导入部分fix_mojibake_escapes = 部分(re.compile(rb'\\u00([\da-f]{2})').sub,lambda m: bytes.fromhex(m.group(1).decode()))使用 open(os.path.join(subdir, file), 'rb') 作为 binary_data:修复 = fix_mojibake_escapes(binary_data.read())数据 = json.loads(repaired.decode('utf8'))

   根据您的示例数据，这会产生:

   {'content': 'No to trzeba ostatnie treningi zrobić xD','sender_name': 'Radosław','时间戳':1524558089，'类型':'通用'}

I downloaded my Facebook messenger data (in your Facebook account, go to settings, then to Your Facebook information, then Download your information, then create a file with at least the Messages box checked) to do some cool statistics

However there is a small problem with encoding. I'm not sure, but it looks like Facebook used bad encoding for this data. When I open it with text editor I see something like this: Rados\u00c5\u0082aw. When I try to open it with python (UTF-8) I get RadosÅ\x82aw. However I should get: Radosław.

My python script:

text = open(os.path.join(subdir, file), encoding='utf-8')
conversations.append(json.load(text))

I tried a few most common encodings. Example data is:

{
  "sender_name": "Rados\u00c5\u0082aw",
  "timestamp": 1524558089,
  "content": "No to trzeba ostatnie treningi zrobi\u00c4\u0087 xD",
  "type": "Generic"
}

解决方案

I can indeed confirm that the Facebook download data is incorrectly encoded; a Mojibake. The original data is UTF-8 encoded but was decoded as Latin -1 instead. I’ll make sure to file a bug report.

In the meantime, you can repair the damage in two ways:

1. Decode the data as JSON, then re-encode any strings as Latin-1, decode again as UTF-8:

   >>> import json
   >>> data = r'"Rados\u00c5\u0082aw"'
   >>> json.loads(data).encode('latin1').decode('utf8')
   'Radosław'
   

2. Load the data as binary, replace all \u00hh sequences with the byte the last two hex digits represent, decode as UTF-8 and then decode as JSON:

   import re
   from functools import partial
   
   fix_mojibake_escapes = partial(
        re.compile(rb'\\u00([\da-f]{2})').sub,
        lambda m: bytes.fromhex(m.group(1).decode()))
   
   with open(os.path.join(subdir, file), 'rb') as binary_data:
       repaired = fix_mojibake_escapes(binary_data.read())
   data = json.loads(repaired.decode('utf8'))
   

   From your sample data this produces:

   {'content': 'No to trzeba ostatnie treningi zrobić xD',
    'sender_name': 'Radosław',
    'timestamp': 1524558089,
    'type': 'Generic'}
   

这篇关于Facebook JSON 编码错误的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！