使用 Twisted 的网络爬虫 [英] Web crawler Using Twisted

问题描述

我正在尝试使用python和twisted创建一个网络爬虫.发生的事情是在调用时reactor.run()

I am trying to create a web crawler with python and twisted.What happend is that at the time of calling reactor.run()

我不知道要获取的所有链接.所以代码如下:

I don't know all the link to get. so the code goes like:

def crawl(url):
    d = getPage(url)
    d.addCallback(handlePage)
    reactor.run()

和句柄页面有类似的东西:

and the handle page has something like:

def handlePage(output):
    urls = getAllUrls(output)

所以现在我需要在 urls 中的每个 url 上应用 crawl().我该怎么做?我应该停止反应器并重新开始吗?如果我遗漏了一些明显的东西，请告诉我.

So now I need to apply the crawl() on each of the url in urls.How do I do that?Should I stop the reactor and start again?If I am missing something obvious please tell me.

推荐答案

您不想停止反应器.您只想下载更多页面.因此，您需要重构您的 crawl 函数，以免停止或启动反应器.

You don't want to stop the reactor. You just want to download more pages. So you need to refactor your crawl function to not stop or start the reactor.

def crawl(url):
    d = getPage(url)
    d.addCallback(handlePage)

def handlePage(output):
    urls = getAllUrls(output)
    for url in urls:
        crawl(url)

crawl(url)
reactor.run()

不过，您可能想查看 scrapy 而不是从头开始构建自己的.

You may want to look at scrapy instead of building your own from scratch, though.

这篇关于使用 Twisted 的网络爬虫的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！