greping 二进制文件和 UTF16 [英] grepping binary files and UTF16

问题描述

标准 grep/pcregrep 等可以方便地用于 ASCII 或 UTF8 数据的二进制文件 - 有没有一种简单的方法让它们也尝试 UTF16(最好同时，但会这样做)?

Standard grep/pcregrep etc. can conveniently be used with binary files for ASCII or UTF8 data - is there a simple way to make them try UTF16 too (preferably simultaneously, but instead will do)?

无论如何，我想要获取的数据都是 ASCII(库中的引用等)，只是找不到，因为有时任何两个字符之间有 00，有时没有.

Data I'm trying to get is all ASCII anyway (references in libraries etc.), it just doesn't get found as sometimes there's 00 between any two characters, and sometimes there isn't.

我看不出有什么方法可以在语义上完成它，但是这些 00 应该可以解决问题，除非我无法在命令行上轻松使用它们.

I don't see any way to get it done semantically, but these 00s should do the trick, except I cannot easily use them on command line.

推荐答案

最简单的方法是将文本文件转换为 utf-8 并将其通过管道传送到 grep:

The easiest way is to just convert the text file to utf-8 and pipe that to grep:

iconv -f utf-16 -t utf-8 file.txt | grep query

我试图做相反的事情(将我的查询转换为 utf-16)，但似乎 grep 不喜欢那样.我认为这可能与字节序有关，但我不确定.

I tried to do the opposite (convert my query to utf-16) but it seems as though grep doesn't like that. I think it might have to do with endianness, but I'm not sure.

似乎 grep 会将 utf-16 的查询转换为 utf-8/ascii.这是我尝试过的:

It seems as though grep will convert a query that is utf-16 to utf-8/ascii. Here is what I tried:

grep `echo -n query | iconv -f utf-8 -t utf-16 | sed 's/..//'` test.txt

如果 test.txt 是 utf-16 文件，这将不起作用，但如果 test.txt 是 ascii，它确实有效.我只能得出结论，grep 正在将我的查询转换为 ascii.

If test.txt is a utf-16 file this won't work, but it does work if test.txt is ascii. I can only conclude that grep is converting my query to ascii.

这是一个非常疯狂的作品，但没有给你很多有用的信息:

Here's a really really crazy one that kind of works but doesn't give you very much useful info:

hexdump -e '/1 "%02x"' test.txt | grep -P `echo -n Test | iconv -f utf-8 -t utf-16 | sed 's/..//' | hexdump -e '/1 "%02x"'`

它是如何工作的?好吧，它将您的文件转换为十六进制(没有 hexdump 通常适用的任何额外格式).它通过管道将其导入 grep.Grep 使用的查询是通过将您的查询(不带换行符)回显到 iconv 来构建的，该 iconv 将其转换为 utf-16.然后将其传送到 sed 以删除 BOM(用于确定字节序的 utf-16 文件的前两个字节).然后将其传送到 hexdump 中，以便查询和输入相同.

How does it work? Well it converts your file to hex (without any extra formatting that hexdump usually applies). It pipes that into grep. Grep is using a query that is constructed by echoing your query (without a newline) into iconv which converts it to utf-16. This is then piped into sed to remove the BOM (the first two bytes of a utf-16 file used to determine endianness). This is then piped into hexdump so that the query and the input are the same.

不幸的是，如果有一个匹配项，我认为这最终会打印出整个文件.如果二进制文件中的 utf-16 存储在与您的机器不同的字节序中，这也将不起作用.

Unfortunately I think this will end up printing out the ENTIRE file if there is a single match. Also this won't work if the utf-16 in your binary file is stored in a different endianness than your machine.

明白了！！！！

grep -P `echo -n "Test" | iconv -f utf-8 -t utf-16 | sed 's/..//' | hexdump -e '/1 "x%02x"' | sed 's/x/\\x/g'` test.txt

这将在文件 test.txt

这篇关于greping 二进制文件和 UTF16的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！