如何使用python和beautifulsoup抓取需要登录的网站? [英] How to scrape a website which requires login using python and beautifulsoup?

问题描述

如果我想抓取一个需要先用密码登录的网站，我如何使用beautifulsoup4库开始用python抓取它?以下是我对不需要登录的网站所做的工作.

If I want to scrape a website that requires login with password first, how can I start scraping it with python using beautifulsoup4 library? Below is what I do for websites that do not require login.

from bs4 import BeautifulSoup    
import urllib2 
url = urllib2.urlopen("http://www.python.org")    
content = url.read()    
soup = BeautifulSoup(content)

应该如何更改代码以适应登录?假设我要抓取的网站是一个需要登录的论坛.一个例子是 http://forum.arduino.cc/index.php

How should the code be changed to accommodate login? Assume that the website I want to scrape is a forum that requires login. An example is http://forum.arduino.cc/index.php

推荐答案

可以使用机械化:

import mechanize
from bs4 import BeautifulSoup
import urllib2 
import cookielib ## http.cookiejar in python3

cj = cookielib.CookieJar()
br = mechanize.Browser()
br.set_cookiejar(cj)
br.open("https://id.arduino.cc/auth/login/")

br.select_form(nr=0)
br.form['username'] = 'username'
br.form['password'] = 'password.'
br.submit()

print br.response().read()

或 urllib - 使用 urllib2 登录网站

Or urllib - Login to website using urllib2

这篇关于如何使用python和beautifulsoup抓取需要登录的网站?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！