如何从需要 cookie 登录的网站中抓取 PHP 中的网站内容? [英] How can I scrape website content in PHP from a website that requires a cookie login?
问题描述
我的问题是它不仅需要基本的 cookie，还需要会话 cookie 和随机生成的 ID.我认为这意味着我需要使用带有 cookie jar 的网络浏览器模拟器?
My problem is that it doesn't just require a basic cookie, but rather asks for a session cookie, and for randomly generated IDs. I think this means I need to use a web browser emulator with a cookie jar?
我曾尝试使用 Snoopy、Goutte 和其他几个网络浏览器模拟器，但到目前为止我还没有找到有关如何接收 cookie 的教程.我有点绝望了！
I have tried to use Snoopy, Goutte and a couple of other web browser emulators, but as of yet I have not been able to find tutorials on how to receive cookies. I am getting a little desperate!
谁能举个例子告诉我如何在 Snoopy 或 Goutte 中接受 cookie?
Can anyone give me an example of how to accept cookies in Snoopy or Goutte?
提前致谢！
推荐答案
面向对象的答案
我们在一个名为 Browser 的类中尽可能多地实现了以前的答案应该提供正常的导航功能.
Object-Oriented answer
We implement as much as possible of the previous answer in one class called Browser that should supply the normal navigation features.
然后我们应该能够以非常简单的形式将特定于站点的代码放在我们称为FooBrowser的新派生类中，该类执行站点的抓取Foo.
Then we should be able to put the site-specific code, in very simple form, in a new derived class that we call, say, FooBrowser, that performs scraping of the site Foo.
派生浏览器的类必须提供一些特定于站点的函数，例如允许存储特定于站点的信息的 path() 函数，例如
The class deriving Browser must supply some site-specific function such as a path() function allowing to store site-specific information, for example
function path($basename) {
return '/var/tmp/www.foo.bar/' . $basename;
}
abstract class Browser
{
private $options = [];
private $state = [];
protected $cookies;
abstract protected function path($basename);
public function __construct($site, $options = []) {
$this->cookies = $this->path('cookies');
$this->options = array_merge(
[
'site' => $site,
'userAgent' => 'Mozilla/5.0 (Windows NT 5.1; rv:16.0) Gecko/20100101 Firefox/16.0 - LeoScraper',
'waitTime' => 250000,
],
$options
);
$this->state = [
'referer' => '/',
'url' => '',
'curl' => '',
];
$this->__wakeup();
}
/**
* Reactivates after sleep (e.g. in session) or creation
*/
public function __wakeup() {
$this->state['curl'] = curl_init();
$this->config([
CURLOPT_USERAGENT => $this->options['userAgent'],
CURLOPT_ENCODING => '',
CURLOPT_NOBODY => false,
// ...retrieving the body...
CURLOPT_BINARYTRANSFER => true,
// ...as binary...
CURLOPT_RETURNTRANSFER => true,
// ...into $ret...
CURLOPT_FOLLOWLOCATION => true,
// ...following redirections...
CURLOPT_MAXREDIRS => 5,
// ...reasonably...
CURLOPT_COOKIEFILE => $this->cookies,
// Save these cookies
CURLOPT_COOKIEJAR => $this->cookies,
// (already set above)
CURLOPT_CONNECTTIMEOUT => 30,
// Seconds
CURLOPT_TIMEOUT => 300,
// Seconds
CURLOPT_LOW_SPEED_LIMIT => 16384,
// 16 Kb/s
CURLOPT_LOW_SPEED_TIME => 15,
]);
}
/**
* Imports an options array.
*
* @param array $opts
* @throws DetailedError
*/
private function config(array $opts = []) {
foreach ($opts as $key => $value) {
if (true !== curl_setopt($this->state['curl'], $key, $value)) {
throw new Exception('Could not set cURL option');
}
}
}
private function perform($url) {
$this->state['referer'] = $this->state['url'];
$this->state['url'] = $url;
$this->config([
CURLOPT_URL => $this->options['site'] . $this->state['url'],
CURLOPT_REFERER => $this->options['site'] . $this->state['referer'],
]);
$response = curl_exec($this->state['curl']);
// Should we ever want to randomize waitTime, do so here.
usleep($this->options['waitTime']);
return $response;
}
/**
* Returns a configuration option.
* @param string $key configuration key name
* @param string $value value to set
* @return mixed
*/
protected function option($key, $value = '__DEFAULT__') {
$curr = $this->options[$key];
if ('__DEFAULT__' !== $value) {
$this->options[$key] = $value;
}
return $curr;
}
/**
* Performs a POST.
*
* @param $url
* @param $fields
* @return mixed
*/
public function post($url, array $fields) {
$this->config([
CURLOPT_POST => true,
CURLOPT_POSTFIELDS => http_build_query($fields),
]);
return $this->perform($url);
}
/**
* Performs a GET.
*
* @param $url
* @param array $fields
* @return mixed
*/
public function get($url, array $fields = []) {
$this->config([ CURLOPT_POST => false ]);
if (empty($fields)) {
$query = '';
} else {
$query = '?' . http_build_query($fields);
}
return $this->perform($url . $query);
}
}
现在抓取 FooSite:
Now to scrape FooSite:
/* WWW_FOO_COM requires username and password to construct */
class WWW_FOO_COM_Browser extends Browser
{
private $loggedIn = false;
public function __construct($username, $password) {
parent::__construct('http://www.foo.bar.baz', [
'username' => $username,
'password' => $password,
'waitTime' => 250000,
'userAgent' => 'FooScraper',
'cache' => true
]);
// Open the session
$this->get('/');
// Navigate to the login page
$this->get('/login.do');
}
/**
* Perform login.
*/
public function login() {
$response = $this->post(
'/ajax/loginPerform',
[
'j_un' => $this->option('username'),
'j_pw' => $this->option('password'),
]
);
// TODO: verify that response is OK.
// if (!strstr($response, "Welcome " . $this->option('username'))
// throw new Exception("Bad username or password")
$this->loggedIn = true;
return true;
}
public function scrape($entry) {
// We could implement caching to avoid scraping the same entry
// too often. Save $data into path("entry-" . md5($entry))
// and verify the filemtime of said file, is it newer than time()
// minus, say, 86400 seconds? If yes, return file_get_content and
// leave remote site alone.
$data = $this->get(
'/foobars/baz.do',
[
'ticker' => $entry
]
);
return $data;
}
现在实际的抓取代码是:
Now the actual scraping code would be:
$scraper = new WWW_FOO_COM_Browser('lserni', 'mypassword');
if (!$scraper->login()) {
throw new Exception("bad user or pass");
}
// www.foo.com is a ticker site, we need little info for each
// Other examples might be much more complex.
$entries = [
'APPL', 'MSFT', 'XKCD'
];
foreach ($entries as $entry) {
$html = $scraper->scrape($entry);
// Parse HTML
}
强制性通知:使用合适的解析器从原始 HTML 中获取数据.
这篇关于如何从需要 cookie 登录的网站中抓取 PHP 中的网站内容?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
