使用python提取网页上的URL列表的简单方法是什么? [英] What is a simple way to extract the list of URLs on a webpage using python?

问题描述

为了好玩，我想创建一个简单的网络爬虫.我需要网络爬虫来获取一页上所有链接的列表.python 库是否有任何内置函数可以使这更容易?感谢任何知识赞赏.

I want to create a simple web crawler for fun. I need the web crawler to get a list of all links on one page. Does the python library have any built in functions that would make this any easier? Thanks any knowledge appreciated.

推荐答案

这其实很简单，BeautifulSoup.

from BeautifulSoup import BeautifulSoup

[element['href'] for element in BeautifulSoup(document_contents).findAll('a', href=True)]

# [u'http://example.com/', u'/example', ...]

最后一件事:你可以使用 urlparse.urljoin 使所有网址成为绝对网址.如果你需要链接文本，你可以使用类似 element.contents[0] 的东西.

One last thing: you can use urlparse.urljoin to make all URLs absolute. If you need the link text, you can use something like element.contents[0].

这里是你如何将它们联系在一起:

And here's how you might tie it all together:

import urllib2
import urlparse
from BeautifulSoup import BeautifulSoup

def get_all_link_targets(url):
    return [urlparse.urljoin(url, tag['href']) for tag in
            BeautifulSoup(urllib2.urlopen(url)).findAll('a', href=True)]

这篇关于使用python提取网页上的URL列表的简单方法是什么?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！