如何使用 Python Scrapy 模块列出我网站上的所有 URL? [英] How do I use the Python Scrapy module to list all the URLs from my website?

问题描述

我想使用 Python Scrapy 模块从我的网站上抓取所有 URL 并将列表写入文件.我查看了示例，但没有看到任何简单的示例来执行此操作.

I want to use the Python Scrapy module to scrape all the URLs from my website and write the list to a file. I looked in the examples but didn't see any simple example to do this.

推荐答案

这是对我有用的 Python 程序:

Here's the python program that worked for me:

from scrapy.selector import HtmlXPathSelector
from scrapy.spider import BaseSpider
from scrapy.http import Request

DOMAIN = 'example.com'
URL = 'http://%s' % DOMAIN

class MySpider(BaseSpider):
    name = DOMAIN
    allowed_domains = [DOMAIN]
    start_urls = [
        URL
    ]

    def parse(self, response):
        hxs = HtmlXPathSelector(response)
        for url in hxs.select('//a/@href').extract():
            if not ( url.startswith('http://') or url.startswith('https://') ):
                url= URL + url 
            print url
            yield Request(url, callback=self.parse)

将其保存在名为 spider.py 的文件中.

Save this in a file called spider.py.

然后您可以使用 shell 管道对文本进行后期处理:

You can then use a shell pipeline to post process this text:

bash$ scrapy runspider spider.py > urls.out
bash$ cat urls.out| grep 'example.com' |sort |uniq |grep -v '#' |grep -v 'mailto' > example.urls

这为我提供了我网站中所有唯一网址的列表.

This gives me a list of all the unique urls in my site.

这篇关于如何使用 Python Scrapy 模块列出我网站上的所有 URL?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！