如何将find命令转换为改用grep过滤，然后在输出中使用`exec`命令? [英] How to convert a find command to instead use grep to filter and then `exec` commands on output?

问题描述

我有一个场景，我需要在找到的每个文件上执行一系列命令.这通常会很好用，除非我有100多个文件和文件夹要从find的执行结果中排除.直接从外壳变得笨拙且不可执行.似乎最好使用类似于tar或grep允许此类文件的方式的排除文件".

I have a scenario where I need to execute a series of commands on each file that's found. This normally would work great, except I have over 100 files and folders to exclude from find's results for execution. This becomes unwieldy and non-executable from the shell directly. It seems like it would be optimal to use an "exclusion file" similar to how tar or grep allows for such files.

由于find不接受排除文件，但是grep接受，我想知道:如何将以下内容转换为可替换排除(prune)和exec函数的命令在find中使用带有排除文件(grep -v -f excludefile)的 grep 来排除文件夹和文件，然后像当前命令那样对结果执行一系列命令:

Since find does not accept a file for exclusion, but grep does, I want to know: how can the following be converted to a command that would replace the exclusion (prune) and exec functions in find to instead utilize grep with an exclusion file (grep -v -f excludefile) to exclude the folders and files and then execute a series of commands on the result like the current command does it:

find $IN_PATH -regextype posix-extended \
  -regex "/(excluded1|excluded2|excluded3|...|excludedN)" -prune \
  -o -type f \
  -exec sh -c "( cmd -with_args 1 '{}'; cmd -args2 '{}'; cmd3 '{}') \
    | cmd4 | cmd5 | cmd6; cmd7 '{}'" \; \
  > output

作为旁注(不是很重要)，我读到，如果您不使用exec，则此过程的效率将大大降低，并且每次运行该过程都将花费100多分钟的时间来执行，因此我不想把它减慢到不必要的程度.

As a side note (not critical), I've read that if you don't use exec this process becomes much less efficient and this process is already consuming over 100 minutes to execute each time that it's run, so I don't want to slow it down any more than is necessary.

推荐答案

我认为要实现您的方案的最佳方法是将单行代码拆分为两行，并引入带有parallel的xargs.

the best way i think of to fulfill your scenario , is split the one-liner to two line and introduce xargs with parallel .

find $IN_PATH -regextype posix-extended \
  -regex "/(excluded1|excluded2|excluded3|...|excludedN)" -prune \
  -o -type f  > /tmp/full_file_list
cat /tmp/full_file_list|grep -f excludefile |xargs -0 -n 1 -P <nr_procs> sh -c 'command here' >output

请参见 Bash脚本并行处理有限数量的命令和在bash中进行并行处理?以了解有关bash中并行的更多信息

see Bash script processing limited number of commands in parallel and Doing parallel processing in bash? to learn more about parallel in bash

对文件的查找和命令在一个衬套中面临disk-io冲突，溢出一个衬套可能会稍微加快该过程，

finding and command on files are facing disk-io conflicts in one liner , spilt the one-liner could speed up the process a little bit ,

提示:请记住将full_file_list/excludefile/output放入排除规则中，并始终在较小的目录上调试命令以减少等待时间

hint: remember to put your full_file_list/excludefile/output in your exclude rules , and always debug your command on a smaller directory to reduce waiting time

这篇关于如何将find命令转换为改用grep过滤，然后在输出中使用`exec`命令?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！