cURL多个同时请求（域检查） [英] cURL Mult Simultaneous Requests (domain check)

问题描述

我想取一份包含20,000个域名的列表，并检查它们是否活着。我真正需要的是一个简单的HTTP代码检查，但我不知道如何使用curl_multi工作。在一个单独的脚本我使用我有以下功能，同时检查一批1000个域，并返回json响应代码。也许这可以修改为只获取http响应代码，而不是页面内容？

I'm trying to take a list of 20,000 + domain names and check if they are "alive". All I really need is a simple http code check but I can't figure out how to get that working with curl_multi. On a separate script I'm using I have the following function which simultaneously checks a batch of 1000 domains and returns the json response code. Maybe this can be modified to just get the http response code instead of the page content?

（抱歉的语法我不能把它粘贴为一个好的块的代码，而不是一行一行，并添加4个空格...（也尝试跳过一行，并添加8个空格）

(sorry about the syntax I couldn't get it to paste as a nice block of code without going line by line and adding 4 spaces...(also tried skipping a line and adding 8 spaces)

$ dotNetRequests =数组的域...

$dotNetRequests = array of domains...

//loop through arrays
foreach(array_chunk($dotNetRequests, 1000) as $Netrequests) {
    $results = checkDomains($Netrequests);
    $NetcurlRequest = array_merge($NetcurlRequest, $results);
}

function checkDomains($data) {

// array of curl handles
$curly = array();
// data to be returned
$result = array();

// multi handle
$mh = curl_multi_init();

// loop through $data and create curl handles
// then add them to the multi-handle
foreach ($data as $id => $d) {

$curly[$id] = curl_init();

$url = (is_array($d) && !empty($d['url'])) ? $d['url'] : $d;
curl_setopt($curly[$id], CURLOPT_URL,            $url);
curl_setopt($curly[$id], CURLOPT_HEADER,         0);
curl_setopt($curly[$id], CURLOPT_RETURNTRANSFER, 1);

// post?
if (is_array($d)) {
  if (!empty($d['post'])) {
    curl_setopt($curly[$id], CURLOPT_POST,       1);
    curl_setopt($curly[$id], CURLOPT_POSTFIELDS, $d['post']);
  }
}

curl_multi_add_handle($mh, $curly[$id]);
  }

  // execute the handles
  $running = null;
  do {
    curl_multi_exec($mh, $running);
  } while($running > 0);

  // get content and remove handles
  foreach($curly as $id => $c) {
     // $result[$id] = curl_multi_getcontent($c);
// if($result[$id]) {
if (curl_multi_getcontent($c)){
    //echo "yes";
    $netName = $data[$id];
    $dName = str_replace(".net", ".com", $netName);
    $query = "Update table1 SET dotnet = '1' WHERE Domain = '$dName'";
    mysql_query($query);
}
curl_multi_remove_handle($mh, $c); 
}

// all done
 curl_multi_close($mh);

return $result;
} 

推荐答案

使用PHP的大量同时cURL请求。
我可以在几分钟内使用它解析5万个域名！

This script works great for handling bulk simultaneous cURL requests using PHP. I'm able to parse through 50k domains in just a few minutes using it!

https://github.com/petewarden/ParallelCurl/

这篇关于cURL多个同时请求（域检查）的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！