用于n核处理器的Perl fork队列 [英] Perl fork queue for n-Core processor

问题描述

我正在编写类似于建议的应用程序

I am writing an application similar to what was suggested here. Essentially, I am using Perl to manage the execution of multiple CPU intensive processes in parallel via fork and wait. However, I am running on a 4-core machine, and I have many more processes, all with very dissimilar expected run-times which aren't known a priori.

与为每个核心简单地使用队列系统相比，最终需要花费更多的精力来估计运行时间并适当地组合它们.最终，我希望每个核心都在尽可能少地停机的情况下进行处理，直到完成所有工作为止.是否有首选的算法或机制可以做到这一点?我认为这是一个常见的问题/用途，所以我不想重新发明轮子，因为我的轮子可能不如正确的方法. '

Ultimately, it would take more effort to estimate the run times and gang them appropriately, than to simply utilize a queue system for each core. Ultimately I want each core to be processing, with as little downtime as possible, until everything is done. Is there a preferred algorithm or mechanism for doing this? I would assume this is a common problem/use so I don't want to re-invent the wheel, as my wheel will probably be inferior to the 'right way. '

作为一个未成年人，我宁愿不必导入其他模块(例如 Parallel :: ForkManager )来完成此操作，但是如果那是最好的方法，那么我会考虑的.

As a minor aside, I would prefer to not have to import additional modules (like Parallel::ForkManager) to accomplish this, but if that is the best way to go, then I will consider it.

〜谢谢！

:修复了此处"链接:感谢 ikegami

Fixed 'here' link: Thanks to ikegami

P :: FM太容易使用了，而不是...我今天学到了.

P::FM is too easy to use, not to... Today I Learned.

推荐答案

Forks::Super 具有一些适合此类任务的功能.

Forks::Super has some features that are good for this sort of task.

• 扩展的语法，但没有很多新语法:如果您已经有一个带有fork和wait调用的程序，则仍然可以使用Forks::Super的功能而无需太多变化.也就是说，您的新代码仍将具有fork和wait调用.
• 作业限制:像Parallel::ForkManager一样，您可以控制同时运行多少个作业.当一项工作完成时，该模块可以启动另一项工作，从而使您的系统得到充分利用.您还可以指定更复杂的逻辑，例如在周末或午夜至凌晨6:00最多运行6个后台作业，而其余时间最多运行2个后台作业"

• 定时实用程序: Forks::Super跟踪每个作业的开始时间和结束时间，让您记录并分析每个作业花费了多长时间:

• extended syntax, but not a lot of new syntax: if you already have a program with fork and wait calls, you can still use the features of Forks::Super without too many changes. That is, your new code will still have fork and wait calls.
• job throttling: like Parallel::ForkManager, you can control how many jobs you run simultaneously. When one job completes, the module can start another one, keeping your system fully utilized. You can also specify more complex logic like "run at most 6 background jobs on the weekends or between midnight and 6:00 am, but 2 background jobs the rest of the time"

• timing utilities: Forks::Super keeps track of the start time and end time of every job, letting you log and analyze how long each job took:

fork { cmd => "some command" };
...
$pid = wait;
$elapsed = $pid->{end} - $pid->{start};
print LOG "That job took ${elapsed}s\n";

CPU相似性控制:我无法确定这是否是您需要的东西，但是Guarav似乎认为这很重要.您可以将后台作业分配给特定的内核

CPU affinity control: I can't tell whether this is something you need, but Guarav seemed to think it mattered. You can assign background jobs to specific cores

# restrict job to cores #0 and #2
$job = fork { sub => \&background_process, args => \@args, 
              cpu_affinity => 0x05 };

这篇关于用于n核处理器的Perl fork队列的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！