如何避免网络爬虫重定向到移动版? [英] How to avoid redirection of the webcrawler to the mobile edition?

问题描述

我子类化了一个 CrawlSpider 并想从网站中提取数据.但是，我总是被重定向到该网站的移动版本.我试图改变将scrapy 设置中的USER_AGENT 变量设置为Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.0.1) Gecko/20060111 Firefox/1.5.0.1，但仍然被重定向.

I subclassed a CrawlSpider and want to extract data from website. However, I always get redirected to the site's mobile version. I tried to change the USER_AGENT variable in scrapy's settings to Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.0.1) Gecko/20060111 Firefox/1.5.0.1, but still get redirected.

是否有另一种方法可以向另一个客户端发出信号并避免重定向?

推荐答案

Scrapy:

• RedirectMiddleware - 根据响应状态处理请求重定向
• MetaRefreshMiddleware - 处理基于元刷新 html 标签的请求重定向

• RedirectMiddleware - Handle redirection of requests based on response status
• MetaRefreshMiddleware - Handle redirection of requests based on meta-refresh html tag

那么，也许您的 html 页面使用了第二种重定向?

So, maybe your html page uses second type of redirection?

另见:

• http://doc.scrapy.org/en/latest/topics/downloader-middleware.html#redirectmiddleware-settings
• http://doc.scrapy.org/en/latest/topics/downloader-middleware.html#metarefreshmiddleware-settings

这篇关于如何避免网络爬虫重定向到移动版?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！