Nutch的：通过把一个cookie的头认证 [英] Nutch: Authentication via putting a cookie in the header

问题描述

我很惊讶有那么一点支持或信息那里获取Nutch的是能够抓取需要身份验证的网站的部分。

I am surprised that there is so little support or information out there for getting Nutch to be able to crawl parts of a website that require authentication.

我知道，也许Apache的Nutch的是目前还无法（但显然希望的）支持HTTP POST验证。

I am aware that maybe Apache Nutch is not currently able to (but apparently hopes to) support Http POST authentication.

但是，我们真正想要做的是能一个cookie添加到我们的Nutch的机器人头部，将允许它访问该网站的部分这种方式（而不是发表用户名和密码的表单，然后接受的cookie）。

However, all we really want to do is be able to add a cookie to our Nutch bot header that will allow it to access those parts of the site that way (rather than post a username and password to a form and then receive the cookie).

所以我花了很多时间量好搜索和很惊讶，这个大多数讨论是早在2005年或2008年一路：的这里，的有，到处。

So I have spent a good amount of time searching and am surprised that most discussions about this are all the way back in 2005 or 2008: here, there, everywhere.

毕竟这些年来，反正是有解决此限制或有刚才还没有办法给出Nutch的一个'prebaked的Cookie，因此它可以访问我们网站的会员只有部分？来验证。

After all these years, is there anyway to work around this limitation or is there just still no way to authenticate by giving Nutch a 'prebaked' cookie so it can access member only parts of our site?.

推荐答案

我添加了自定义的code到Nutch的协议HttpClient的插件来解决这个问题。

I have added custom code to nutch protocol-httpclient plugin to solve the issue.

下面的共享

<一个href=\"http://www.gingercart.com/Home/search-and-crawl/nutch-custom-authentication-cookies-session-management-to-crawl-secure-enterprise-websites\" rel=\"nofollow\">http://www.gingercart.com/Home/search-and-crawl/nutch-custom-authentication-cookies-session-management-to-crawl-secure-enterprise-websites

这篇关于Nutch的：通过把一个cookie的头认证的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！