grep多个模式单个文件参数列表太长 [英] grep multiple patterns single file argument list too long

问题描述

我当前正在文件中搜索多个模式.该文件的大小为90GB，我在特定字段中搜索(每行6-17位).我试图获取包含任何特定数字列表的所有行.我正在使用的当前语法是:

I am currently searching for multiple patterns in a file. The file is of 90GB in size, I am searching on a particular field(from position 6-17 in each line). I am trying to get all the lines that contain any of a particular list of numbers. The current syntax I am using is:

grep '^.\{6\}0000000012345\|^.\{6\}0000000012543' somelargeFile.txt > outputFile.txt

对于少量模式，此方法有效.对于大量模式，我收到参数列表过长"错误.

For small number of patterns this works. For a large number of patterns I get the "Argument list too long" error.

我尝试过的另一种方法是分别搜索每个模式(在模式上使用for循环)，但这将需要对大型数据文件(57102722行)进行多次遍历，这效率不高.

One alternative I have tried is to search for each patters separately (using a for loop over the patterns), but this will require multiple passes over the large data file(57102722 lines) which is not efficient.

根据我对参数列表过长"错误的了解，它通常与bash cmds有关，而与grep无关.是否有任何设置可用于解决此错误?或者，关于使用awk或sed或其他工具执行此操作的任何想法?

From what I understand about the "Argument list too long" error, it is related to bash cmds in general and not specific to grep. Is there any setting that can be used to get around this error? Or alternatively, any ideas as to how to do this using awk or sed or another tool?

谢谢！

推荐答案

您可以通过将模式放在文件中，并对grep使用 -f 命令行选项来避免此问题.

You can avoid the problem by putting the patterns in a file, and using the -f command line option to grep.

最方便的方法是将每个替代项放在文件的单独行中:

The most convenient is to put each alternative in a separate line of the file:

^.\{6\}0000000012345
^.\{6\}0000000012543

调用

grep -f patterns.txt somelargeFile.txt > outputFile.txt

这篇关于grep多个模式单个文件参数列表太长的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！