如何防止在刮擦亚马逊时被列入黑名单 [英] How to prevent getting blacklisted while scraping Amazon

问题描述

我试图用Scrapy刮擦亚马逊。
但我有此错误

I try to scrape Amazon by Scrapy. but i have this error

DEBUG: Retrying <GET http://www.amazon.fr/Amuses-bouche-Peuvent-b%C3%A9n%C3%A9ficier-dAmazon-Premium-Epicerie/s?ie=UTF8&page=1&rh=n%3A6356734031%2Cp_76%3A437878031> 
(failed 1 times): 503 Service Unavailable

我认为这是因为= Amazon非常擅长检测机器人。
我该如何预防？

I think that it's because = Amazon is very good at detecting bots. How can i prevent this?

我在每次请求前都使用了 time.sleep（6）。

i used time.sleep(6) before every request.

我不想使用他们的API。

I don't want to use their API.

我尝试使用tor和策略

推荐答案

在使用Amazon时必须非常小心，并遵守Amazon条款与网络抓取有关的使用和政策。

You have to be very careful with Amazon and follow the Amazon Terms of Use and policies related to web-scraping.

亚马逊非常擅长禁止漫游器的IP。您必须调整 DOWNLOAD_DELAY 和 CONCURRENT_REQUESTS 可以减少访问该网站的频率，并成为一名良好的爬网公民。而且，您需要旋转IP地址（例如，您可以查看 crawlera ）和用户代理。

Amazon is quite good at banning IPs of the bots. You would have to tweak the DOWNLOAD_DELAY and CONCURRENT_REQUESTS to hit the website less often and be a good web-scraping citizen. And, you would need to rotate IP addresses (you may look into, for instance, crawlera) and user agents.

这篇关于如何防止在刮擦亚马逊时被列入黑名单的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！