grepping二进制文件和UTF16 [英] grepping binary files and UTF16

问题描述

标准 grep / pcregrep 等可方便地用于ASCII或UTF8数据的二进制文件 - 有没有一个简单的方法，使他们尝试UTF16也（最好是同时，但将做）？

数据我试图得到的是所有的ASCII无论如何（库中的参考等），它只是没有被发现，因为有时有两个字符之间有00，有时不存在。

我没有看到任何方式获得它在语义上完成了，但是这些00s应该做到这一点，除非我不能在命令行中轻松使用它们。 解决方案

最简单的方法只是将文本文件转换为utf-8并将其转换为grep：

  iconv -f utf-16 -t utf -8 file.txt | grep查询
  

我试图做相反的事情（将我的查询转换为utf-16）虽然grep不喜欢那样。我认为这可能与endianness有关，但我不确定。

好像grep会将一个查询转换为utf-16到utf-8 / ASCII。这是我试过的：

  grep`echo -n query | iconv -f utf-8 -t utf-16 | sed's /..//'test.txt 
  

如果test.txt是一个utf- 16文件这将无法正常工作，但如果test.txt是ascii，它就可以工作。我只能得出结论，grep正在将我的查询转换为ascii。

编辑：这是一个非常疯狂的工作，但不会给你非常有用的信息：

  hexdump -e'/ 1％02x'test.txt | grep -P`echo -n Test | iconv -f utf-8 -t utf-16 | sed's /..//'| hexdump -e'/ 1％02x``
  

它是如何工作的？那么它将您的文件转换为十六进制（不需要任何额外的格式，通常使用hexdump）。它将其转换为grep。 Grep使用通过将你的查询（不带换行符）回显到iconv中的查询来将它转换为utf-16。然后将其输入sed以删除BOM（用于确定排序的utf-16文件的前两个字节）。然后将它传送到hexdump中，以便查询和输入相同。

不幸的是，我认为如果只有一个匹配，这将最终打印整个文件。如果你的二进制文件中的utf-16存储在与你的机器不同的字节顺序中，这也是行不通的。

编辑2：明白了!!!!

p>

  grep -P`echo -nTest| iconv -f utf-8 -t utf-16 | sed's /..//'| hexdump -e'/ 1x％02x'| sed's / x / \\\x / g'` test.txt 
  

在文件 test.txt 中搜索字符串 Test （在utf-16中）的十六进制版本

Standard grep/pcregrep etc. can conveniently be used with binary files for ASCII or UTF8 data - is there a simple way to make them try UTF16 too (preferably simultaneously, but instead will do)?

Data I'm trying to get is all ASCII anyway (references in libraries etc.), it just doesn't get found as sometimes there's 00 between any two characters, and sometimes there isn't.

I don't see any way to get it done semantically, but these 00s should do the trick, except I cannot easily use them on command line.

解决方案

The easiest way is to just convert the text file to utf-8 and pipe that to grep:

iconv -f utf-16 -t utf-8 file.txt | grep query

I tried to do the opposite (convert my query to utf-16) but it seems as though grep doesn't like that. I think it might have to do with endianness, but I'm not sure.

It seems as though grep will convert a query that is utf-16 to utf-8/ascii. Here is what I tried:

grep `echo -n query | iconv -f utf-8 -t utf-16 | sed 's/..//'` test.txt

If test.txt is a utf-16 file this won't work, but it does work if test.txt is ascii. I can only conclude that grep is converting my query to ascii.

EDIT: Here's a really really crazy one that kind of works but doesn't give you very much useful info:

hexdump -e '/1 "%02x"' test.txt | grep -P `echo -n Test | iconv -f utf-8 -t utf-16 | sed 's/..//' | hexdump -e '/1 "%02x"'`

How does it work? Well it converts your file to hex (without any extra formatting that hexdump usually applies). It pipes that into grep. Grep is using a query that is constructed by echoing your query (without a newline) into iconv which converts it to utf-16. This is then piped into sed to remove the BOM (the first two bytes of a utf-16 file used to determine endianness). This is then piped into hexdump so that the query and the input are the same.

Unfortunately I think this will end up printing out the ENTIRE file if there is a single match. Also this won't work if the utf-16 in your binary file is stored in a different endianness than your machine.

EDIT2: Got it!!!!

grep -P `echo -n "Test" | iconv -f utf-8 -t utf-16 | sed 's/..//' | hexdump -e '/1 "x%02x"' | sed 's/x/\\\\x/g'` test.txt

This searches for the hex version of the string Test (in utf-16) in the file test.txt

这篇关于grepping二进制文件和UTF16的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！