python请求机器人检测? [英] python requests bot detection?
问题描述
我一直在使用 requests 库来挖掘这个网站.我没有在 10 分钟内向它提出太多要求.说 25.突然，网站给了我 404 错误.
I have been using the requests library to mine this website. I haven't made too many requests to it within 10 minutes. Say 25. All of a sudden, the website gives me a 404 error.
我的问题是:我在某处读到过，使用浏览器获取 URL 与使用诸如 requests 之类的内容获取 URL 不同.因为 requests fetch 不会获取浏览器会获取的 cookie 和其他东西.requests 中是否有选项可以模拟浏览器，以便服务器不会认为我是机器人?或者这不是问题?
My question is: I read somewhere that getting a URL with a browser is different from getting a URL with something like a requests. Because the requests fetch does not get cookies and other things that a browser would. Is there an option in requests to emulate a browser so the server doesn't think i'm a bot? Or is this not an issue?
推荐答案
基本上，您至少可以做的一件事就是发送 User-Agent 标头:
Basically, at least one thing you can do is to send User-Agent header:
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:20.0) Gecko/20100101 Firefox/20.0'}
response = requests.get(url, headers=headers)
除了 requests 之外，您还可以使用 selenium 模拟真实用户- 它使用真正的浏览器 - 在这种情况下，显然没有简单的方法可以将您的自动化用户与其他用户区分开来.Selenium 还可以使用无头"浏览器.
Besides requests, you can simulate a real user by using selenium - it uses a real browser - in this case there is clearly no easy way to distinguish your automated user from other users. Selenium can also make use a "headless" browser.
另外，检查您正在抓取的网站是否提供了 API.如果没有 API 或者您没有使用它，请确保您知道该站点是否真的允许像这样自动进行网络爬行，研究使用条款.您知道，他们在一段时间内收到过多请求后阻止您可能是有原因的.
Also, check if the web-site you are scraping provides an API. If there is no API or you are not using it, make sure you know if the site actually allows automated web-crawling like this, study Terms of use. You know, there is probably a reason why they block you after too many requests per a period of time.
另见:
edit1:selenium 使用 webdriver 而不是真正的浏览器；即，它在标头中传递 webdriver = TRUE，使其比 requests 更容易检测.
edit1: selenium uses a webdriver rather than a real browser; i.e., it passes a webdriver = TRUE in the header, making it far easier to detect than requests.
这篇关于python请求机器人检测?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
