Gnu Parallel:嵌套并行 [英] Gnu Parallel : nested parallelism

问题描述

是否可以在由 gnu parallel 生成的脚本的多次运行中调用 gnu parallel?

Is it possible to call gnu parallel from within multiple runs of a script that are in-turn spawned by gnu parallel?

我有一个 python 脚本，可以运行 100 次 顺序 迭代，并且在每次迭代中的某个地方，并行计算 4 个值(使用 gnu 并行).现在我想再次使用 gnu parallel 同时生成多个这样的脚本.这可能吗?gnu parallel 会很好地利用可用内核吗?

I have a python script that runs for 100s of sequential iterations, and somewhere within each iteration, 4 values are being computed in parallel (using gnu parallel). Now I want to spawn multiple such scripts at the same time, again, using gnu parallel. Is this possible? Will gnu parallel take care of good utilization of available cores?

例如，如果在内循环中，4 个值中有 2 个已完成，2 个正在运行，因此在计算所有 4 个值之前，单个脚本无法进行下一次迭代.两个空闲内核是否会自动用于计算不同脚本运行的结果?如何指定可用内核的总数?在内部调用并行还是外部调用?

For example, if in the inner loop, out of 4 values, 2 have been completed and 2 are running, so that a single script cannot proceed to the next iteration until all 4 values are computed. Will the two free cores be used for computing results for a different run of the script automatically? How can I specify the total number of cores available? In the inner call to parallel or outer call?

这个问题表明可以将调用嵌套到并行，但是当我从脚本内部调用嵌套并行时，我不确定这是否会改变.

This questions shows it is possible to nest calls to parallel, but I'm not sure if this changes when I'm calling the nested parallel from inside a script.

PS:抖动不是问题，我可以使用大型集群中的大量内核.

PS: Thrashing is not a concern, I can use a LOT of cores from a large cluster.

PS2:gnu-parallel 是一个很棒的工具……谢谢！:)

PS2: gnu-parallel is an AWESOME tool... thanks! : )

推荐答案

是的.GNU Parallel 旨在(并经过大量测试)能够从 GNU Parallel 调用 - 直接或通过脚本调用.

Yes. GNU Parallel is designed (and tested heavily) to be able to be called from GNU Parallel - either directly or through a script.

如果直接调用，您可能需要更改-I.第二个 {} 在这里是什么意思(它指的是第一个还是第二个平行?):

If called directly you are likely to need to change -I. What does the second {} mean here (does it refer to the first or the second parallel?):

seq 10 | parallel 'seq {} | parallel echo {}'

这里很清楚:

seq 10 | parallel 'seq {} | parallel -I // echo //'
seq 10 | parallel 'seq {} | parallel -I // echo {} //'

PS 很高兴听到您发现 GNU Parallel 很棒.如果您喜欢 GNU Parallel:

PS Good to hear you find GNU Parallel awesome. If you like GNU Parallel:

• 完成教程(http://www.gnu.org/software/parallel/parallel_tutorial.html)
• 在您当地的用户组/团队/同事中进行演示
• 在 Reddit/Diaspora*/forums/blogs/上发布介绍视频和教程Identi.ca/Google+/Twitter/Facebook/Linkedin/邮件列表
• 为您喜爱的博客或杂志请求或撰写评论
• 邀请我参加您的下一次会议

如果您使用 GNU Parallel 进行研究:

If you use GNU Parallel for research:

• 请在您的出版物中引用 GNU Parallel(使用 --bibtex)

如果 GNU Parallel 为您省钱:

If GNU Parallel saves you money:

• (有你的公司)向 FSF 捐款 https://my.fsf.org/donate/

这篇关于Gnu Parallel:嵌套并行的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！