CURL问题（多） [英] Problem with CURL (Multi)

问题描述

我遇到 curl_multi _ * ，我想创建一个类/函数接收，让我们说1000个URL，并一次处理所有这些URL 5，所以当一个URL完成下载它会分配现在可用的插槽到一个新的URL，没有已处理完毕。

I'm having a problem with curl_multi_*, I want to create a class / function that receives, lets say 1000 URLs, and processes all those URLs 5 at a time, so when a URL finishes downloading it will allocate the now available slot to a new URL that hasn't been processed yet.

我看过 some 实施的curl_multi，但没有一个允许我做我想要的，我相信解决方案在某处使用 curl_multi_select 但文档不是很清楚，用户注释没有帮助。

I've seen some implementations of curl_multi, but none of them allows me to do what I want, I believe the solution lies somewhere in the usage of curl_multi_select but the documentation isn't very clear and the user notes don't help much.

任何人都可以向我提供一些例子，说明如何实现这样的功能。

Can anyone please provide me with some examples how I can implement such a feature?

推荐答案

。此脚本会一次获取任意数量的网址，并在每个网址完成后添加一个新网址（因此它总是获取 $ maxConcurrent 页）。

Here's one way to do it. This script will fetch any number of urls at a time, and add a new one as each is finished (so it's always fetching $maxConcurrent pages).

$sites = array('http://example.com', 'http://google.com', 'http://stackoverflow.com');
$concurrent = 2;   // Any number.

$mc = new MultiCurl($sites, $concurrent);
$mc->process();

echo '</pre>';

class MultiCurl
{
    private $allToDo;
    private $multiHandle;
    private $maxConcurrent = 2;
    private $currentIndex  = 0;
    private $info          = array();
    private $options       = array(CURLOPT_RETURNTRANSFER => true,
                                   CURLOPT_FOLLOWLOCATION => true,
                                   CURLOPT_MAXREDIRS      => 3,
                                   CURLOPT_TIMEOUT        => 3);

    public function __construct($todo, $concurrent)
    {
        $this->allToDo = $todo;
        $this->maxConcurrent = $concurrent;
        $this->multiHandle = curl_multi_init();
    }

    public function process()
    {
        $running = 0;
        do {
            $this->_addHandles(min(array($this->maxConcurrent - $running, $this->_moreToDo())));
            while ($exec = curl_multi_exec($this->multiHandle, $running) === -1) {
            }
            curl_multi_select($this->multiHandle);
            while ($multiInfo = curl_multi_info_read($this->multiHandle, $msgs)) {
                $this->_showData($multiInfo);
                curl_multi_remove_handle($this->multiHandle, $multiInfo['handle']);
                curl_close($multiInfo['handle']);
            }
        } while ($running || $this->_moreTodo());
        return $this;
    }    

    private function _addHandles($num)
    {
        while ($num-- > 0) {
            $handle = curl_init($this->allToDo[$this->currentIndex]);
            curl_setopt_array($handle, $this->options);
            curl_multi_add_handle($this->multiHandle, $handle);
            $this->info[$handle]['url'] = $this->allToDo[$this->currentIndex];
            $this->currentIndex++;
        }
    }        

    private function _moreToDo()
    {
        return count($this->allToDo) - $this->currentIndex;
    }

    private function _showData($multiInfo)
    {
        $this->info[$multiInfo['handle']]['multi'] = $multiInfo;
        $this->info[$multiInfo['handle']]['curl']  = curl_getinfo($multiInfo['handle']);
        //print_r($this->info[$multiInfo['handle']]);
        $content = curl_multi_getcontent($multiInfo['handle']);
        echo $this->info[$multiInfo['handle']]['url'] . ' - ' . strlen($content) . ' bytes<br />';
        //echo htmlspecialchars($content);
    }
}

这篇关于CURL问题（多）的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！