在浏览器中找到页面时，cURL 返回 404 [英] cURL returns 404 while the page is found in browser

问题描述

stackoverflow 上已经有类似的问题，但他们的解决方案都没有对我有用.我正在尝试使用 cURL 在 LoveIt.com 上抓取一个页面，但它返回 404 错误，而该 url 在浏览器中工作正常:

there is already similar questions on stackoverflow, but none of their solutions have been working for me. I'm trying to grab a page on LoveIt.com with cURL, but it returns me a 404 error, while the url works fine in the browser :

        $url = 'http://loveit.com/loves/P0D1jlFaIOzzZfZqj_bY3KV';

        $curl = curl_init();
        curl_setopt($curl, CURLOPT_URL, $url);
        curl_setopt($curl, CURLOPT_USERAGENT, "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)");
        curl_setopt ($curl, CURLOPT_HEADER, false);
        curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);
        curl_setopt($curl, CURLOPT_FOLLOWLOCATION, true);
        curl_setopt($curl, CURLOPT_REFERER,'http://loveit.com/');

这是我收到的标题:

数组 ( [url] => http://loveit.com/loves/P0D1jlFaIOzzZfZqj_bY3KV [content_type] => text/html; charset=utf-8 [http_code] => 404 [header_size] => 667 [request_size] => 172 [filetime] => -1 [ssl_verify_result] => 0 [redirect_count] => 0 [total_time] => 0.320466 [namelookup_time] => 0.000326 [connect_time] => 0.119046 [pretransfer_time] => 0.119089 [size_upload] => 0 [size_download] => 499 [speed_load57] => 15_up_length=> 499 [upload_content_length] => 0 [starttransfer_time] => 0.320438 [redirect_time] => 0 [certinfo] => Array () [primary_ip] => --- [primary_port] => 80 [local_ip] => --- [local_port] => 53837 [redirect_url] => )

Array ( [url] => http://loveit.com/loves/P0D1jlFaIOzzZfZqj_bY3KV [content_type] => text/html; charset=utf-8 [http_code] => 404 [header_size] => 667 [request_size] => 172 [filetime] => -1 [ssl_verify_result] => 0 [redirect_count] => 0 [total_time] => 0.320466 [namelookup_time] => 0.000326 [connect_time] => 0.119046 [pretransfer_time] => 0.119089 [size_upload] => 0 [size_download] => 499 [speed_download] => 1557 [speed_upload] => 0 [download_content_length] => 499 [upload_content_length] => 0 [starttransfer_time] => 0.320438 [redirect_time] => 0 [certinfo] => Array ( ) [primary_ip] => --- [primary_port] => 80 [local_ip] => --- [local_port] => 53837 [redirect_url] => )

我读到一些网站对此类脚本有保护措施；我确实测试了一些提出的解决方案，但没有一个对我有用(CURLOPT_USERAGENT，CURLOPT_REFERER ...)

I read that some website had protections against this kind of scripts; and I did test some solutions proposed, but none worked for me (CURLOPT_USERAGENT,CURLOPT_REFERER...)

对这里发生的事情有任何想法吗?

Any ideas of what's happening here ?

我想备份我的 LoveIt 帐户，这就是我做这个的原因(没有导出功能，也没有 LoveIt.com 对网站健康状况的回复)

I would like to backup my LoveIt account, that's why i'm making this (no exports functions and no replies from LoveIt.com about the health of the website)

推荐答案

我在启用 LiveHeaders 的情况下快速检查了上述页面，发现设置了一堆 cookie.我怀疑，因为它不是正常"的 url，你需要在重定向时传递这些 cookie，否则你最终会被 404 踢出.在开始时使用 CURLOPT_COOKIEJAR 和你的 cURL 实例.请参阅:http://php.net/manual/pl/function.curl-setopt.php

I quickly checked the said page with LiveHeaders enabled and I noticed bunch of cookies set. I suspect that, since it's not "normal" url, you need to hand those cookies while being redirected otherwise you end being kicked out with 404. Use CURLOPT_COOKIEJAR with your cURL instance at start. See: http://php.net/manual/pl/function.curl-setopt.php

这篇关于在浏览器中找到页面时，cURL 返回 404的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！