如何有效地使用grep？ [英] How to use grep efficiently?

问题描述

我有大量的小文件需要搜索。我一直在寻找一个很好的事实上的多线程版本 grep ，但找不到任何东西。我该如何改进我对grep的使用？截至目前，我正在这样做：

  grep -Rstring>>字符串
  

解决方案

如果您在多核上安装了xargs处理器，您可以从以下方面受益，以防万一有人感兴趣。

环境：

 处理器：双核四核2.4GHz 
内存：32 GB 
文件数量：584450 
总大小：〜35 GB 
  

测试：

找到必要的文件，将它们管到xargs并告诉它执行8个实例。

名称* .ext-print0 | xargs -0 -n1 -P8 grep -Hstring>> Strings_find8

real 3m24.358s
user 1m27.654s
sys 9m40.316s

2。找到必要的文件，将它们管到xargs并告诉它执行4个实例。

名称* .ext-print0 | xargs -0 -n1 -P4 grep -Hstring>>字符串

真实16m3.051s
用户0m56.012s
sys 8m42.540s

3。建议者@Stephen：找到必要的文件，并用+代替xargs

  time find ./ -name * .ext-exec grep -Hstring{} \ +>>字符串
 
真实53m45.438s 
用户0m5.829s 
 sys 0m40.778s 
  

4。定期递归grep。

  grep -Rstring>>字符串
 
真实235m12.823s 
用户38m57.763s 
 sys 38m8.301s 
  

出于我的目的，第一个命令工作得很好。

I have a large number of small files to be searched. I have been looking for a good de-facto multi-threaded version of grep but could not find anything. How can I improve my usage of grep? As of now I am doing this:

grep -R "string" >> Strings

解决方案

If you have xargs installed on a multi-core processor, you can benefit from the following just in case someone is interested.

Environment:

Processor: Dual Quad-core 2.4GHz
Memory: 32 GB
Number of files: 584450
Total Size: ~ 35 GB

Tests:

1. Find the necessary files, pipe them to xargs and tell it to execute 8 instances.

time find ./ -name "*.ext" -print0 | xargs -0 -n1 -P8 grep -H "string" >> Strings_find8

real    3m24.358s
user    1m27.654s
sys     9m40.316s

2. Find the necessary files, pipe them to xargs and tell it to execute 4 instances.

time find ./ -name "*.ext" -print0 | xargs -0 -n1 -P4 grep -H "string" >> Strings

real    16m3.051s
user    0m56.012s
sys     8m42.540s

3. Suggested by @Stephen: Find the necessary files and use + instead of xargs

time find ./ -name "*.ext" -exec grep -H "string" {} \+ >> Strings

real    53m45.438s
user    0m5.829s
sys     0m40.778s

4. Regular recursive grep.

grep -R "string" >> Strings

real    235m12.823s
user    38m57.763s
sys     38m8.301s

For my purposes, the first command worked just fine.

这篇关于如何有效地使用grep？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！