从字符串中删除每个非utf-8符号 [英] Delete every non utf-8 symbols from string
本文介绍了从字符串中删除每个非utf-8符号的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!
问题描述
我有大量的文件和解析器.我要做的是剥离所有非utf-8符号并将数据放入mongodb中. 目前,我有这样的代码.
I have a big amount of files and parser. What I Have to do is strip all non utf-8 symbols and put data in mongodb. Currently I have code like this.
with open(fname, "r") as fp:
for line in fp:
line = line.strip()
line = line.decode('utf-8', 'ignore')
line = line.encode('utf-8', 'ignore')
我仍然会收到错误消息
bson.errors.InvalidStringData: strings in documents must be valid UTF-8:
1/b62010montecassianomcir\xe2\x86\x90ta0\xe2\x86\x90008923304320733/290066010401040101506055soccorin
我不明白.有一些简单的方法可以做到吗?
I don't get it. Is there some simple way to do it?
UPD :似乎Python和Mongo对Utf-8有效字符串的定义并不相同.
UPD: seems like Python and Mongo don't agree about definition of Utf-8 Valid string.
推荐答案
请尝试在下面的代码行而不是最后两行.希望对您有所帮助:
Try below code line instead of last two lines. Hope it helps:
line=line.decode('utf-8','ignore').encode("utf-8")
这篇关于从字符串中删除每个非utf-8符号的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!
查看全文