使用并行方式调用 shell 函数，并使用带引号的文件名列表作为输入 [英] calling shell function using parallel with list of quoted filenames as input

问题描述

使用 Bash.

我有一个导出的 shell 函数，我想将其应用于许多文件.

I have an exported shell function which I want to apply to many files.

通常我会使用 xargs，但语法是这样的(参见 此处)太难看了.

Normally I would use xargs, but the syntax like this (see here) is too ugly for use.

<代码>...... |xargs -n 1 -P 10 -I {} bash -c 'echo_var "$@"' _ {}

在那次讨论中，parallel 有一个更简单的语法:

In that discussion, parallel had an easier syntax:

<代码>..... |并行 -P 10 echo_var {}

现在我遇到了以下问题:我想应用我的函数的文件列表是一行上的文件列表，每个文件都用空格引用和分隔，因此:文件1"文件2"文件3".

Now I have run into the following problem: the list of files to which I want to apply my function is a list of files on one line, each quoted and separated by spaces thus: "file 1" "file 2" "file 3".

我怎样才能将这个空格分隔的、引用的、列表输入到 parallel 中?

how can I feed this space-separated, quoted, list into parallel?

我可以使用 echo 复制列表进行测试.

I can replicate the list using echo for testing.

例如

echo '"file 1" "file 2" "file 3"'|parallel -d " " my_function {}

但我无法让它工作.

我该如何解决?

推荐答案

我该如何解决?

您必须选择一个唯一的分隔符.

You have to choose a unique separator.

echo 'file 1|file 2|file 3' | xargs -d "|" -n1 bash -c 'my_function "$@"' --
echo 'file 1^file 2^file 3' | parallel -d "^" my_function

最安全的是使用零字节作为分隔符:

The safest is to use zero byte as the separator:

echo -e 'file 1\x00file 2\x00file 3' | xargs -0 ' -n1 bash -c 'my_function "$@"' --
printf "%s\0" 'file 1' 'file 2' 'file 3' | parallel -0 my_function

最好是将元素存储在 bash 数组中并使用零分隔流来处理它们:

The best is to store your elements inside a bash array and use a zero separated stream to process them:

files=("file 1" "file 2" "file 3")
printf "%s\0" "${files[@]}" | xargs -0 -n1 bash -c 'my_function "$@"' --
printf "%s\0" "${files[@]}" | parallel -0 my_function

请注意，空数组将在没有任何参数的情况下运行该函数.有时更喜欢使用 -r --no-run-if-empty 选项在输入为空时不运行该函数.--no-run-if-empty 由 parallel 支持，并且是 xargs (xargs> 在 BSD 和 OSX 上没有 --no-run-if-empty).

Note that empty arrays will run the function without any arguments. It's sometimes preferred to use -r --no-run-if-empty option not to run the function when input is empty. The --no-run-if-empty is supported by parallel and is a gnu extension in xargs (xargs on BSD and on OSX do not have --no-run-if-empty).

注意:xargs 默认解析 '、" 和 \.这就是为什么以下是可能的并将工作:

Note: xargs by default parses ', " and \. This is why the following is possible and will work:

echo '"file 1" "file 2" "file 3"' | xargs -n1 bash -c 'my_function "$@"' --
echo "'file 1' 'file 2' 'file 3'" | xargs -n1 bash -c 'my_function "$@"' --
echo 'file\ 1 file\ 2 file\ 3' | xargs -n1 bash -c 'my_function "$@"' --

它可能会导致一些奇怪的事情，所以记住几乎总是为 xargs 指定 -d 选项:

And it can result in some strange things, so remember to almost always specify -d option to xargs:

$ # note \x replaced by single x
$ echo '\\a\b\c' | xargs
\abc
$ # quotes are parsed and need to match
$ echo 'abc"def' | xargs
xargs: unmatched double quote; by default quotes are special to xargs unless you use the -0 option
$ echo "abc'def" | xargs
xargs: unmatched single quote; by default quotes are special to xargs unless you use the -0 option

xargs 是一个随处可用的便携式工具，而 parallel 是一个 GNU 程序，必须单独安装.

xargs is a portable tool available quite everywhere, while parallel is a GNU program, which has to be installed separately.

这篇关于使用并行方式调用 shell 函数，并使用带引号的文件名列表作为输入的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！