使用线程的Perl并行请求 [英] Perl parallel requests using threads

问题描述

我找到了答案解决方案

要启动的线程很多.它可能低于您的系统的系统线程限制，因此它取决于您有多少资源可用于这项工作.

如果您想使用工作池，则 Parallel:ForkManager 很受欢迎该模块.

该模块的文档为大量下载者提供了此示例:

use LWP::Simple;
use Parallel::ForkManager;

...

@links=(
  ["http://www.foo.bar/rulez.data","rulez_data.txt"],
  ["http://new.host/more_data.doc","more_data.doc"],
  ...
);

...

# Max 30 processes for parallel download
my $pm = Parallel::ForkManager->new(30);

foreach my $linkarray (@links) {
  $pm->start and next; # do the fork

  my ($link,$fn) = @$linkarray;
  warn "Cannot get $fn from $link"
    if getstore($link,$fn) != RC_OK;

  $pm->finish; # do the exit in the child process
}
$pm->wait_all_children;

LWP :: UserAgent 没有与<一个href ="https://metacpan.org/pod/LWP::Simple" rel ="nofollow noreferrer"> LWP :: Simple 提供，但是它确实有一个mirror方法，其行为类似. /p>

i found an answer here on using threads for http requests

Just want to ask, in the accepted answer:

for my $url ('http://www.google.com/', 'http://www.perl.org/') {
   push @threads, async { $ua->get($url) };
}

if i have more than 20K urls to fetch, is this approach of pushing to array @threads inside this for loop advisable? Or should i restructure it to handle more than 20K list items? How can i do it such that it doesn't crash my system? thanks

解决方案

That is quite a few threads to launch. It's probably below the thread limit for your system, so it depends how many resources you have available for the job.

If you'd rather use a worker pool, Parallel:ForkManager is a popular module for that.

The module's documentation offers this example for a mass-downloader:

use LWP::Simple;
use Parallel::ForkManager;

...

@links=(
  ["http://www.foo.bar/rulez.data","rulez_data.txt"],
  ["http://new.host/more_data.doc","more_data.doc"],
  ...
);

...

# Max 30 processes for parallel download
my $pm = Parallel::ForkManager->new(30);

foreach my $linkarray (@links) {
  $pm->start and next; # do the fork

  my ($link,$fn) = @$linkarray;
  warn "Cannot get $fn from $link"
    if getstore($link,$fn) != RC_OK;

  $pm->finish; # do the exit in the child process
}
$pm->wait_all_children;

LWP::UserAgent doesn't have the same getstore sub that LWP::Simple provides, but it does have a mirror method which behaves similarly.

这篇关于使用线程的Perl并行请求的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！