一些网站阻止了硒webdriver，这是如何工作的? [英] Some websites block selenium webdriver, how does this work?

问题描述

因此，我尝试通过Web搜寻服装网站来构建要寻找的超值优惠/产品列表，但我注意到我尝试加载的某些网站没有.网站如何阻止Selenium Webdriver http请求?他们在看标题还是其他东西.您能否逐步介绍Selenium Webdriver如何发送请求以及服务器如何接收请求/如何阻止请求?

So I'm trying to web crawl clothing websites to build a list of great deals/products to look out for, but I notice that some of the websites that I try to load, don't. How are websites able to block selenium webdriver http requests? Do they look at the header or something. Can you give me a step by step of how selenium webdriver sends requests and how the server receives them/ are able to block them?

推荐答案

Selenium使用真正的Web浏览器(通常是Firefox或Chrome)发出请求，因此该网站可能不知道您在后台使用Selenium.

Selenium uses a real web browser (typically Firefox or Chrome) to make its requests, so the website probably has no idea that you're using Selenium behind the scenes.

如果该网站阻止了您，则可能是由于您的使用方式(即您每分钟发出1000个请求而阻塞了他们的网络服务器.这很不礼貌.不要这样做！)

If the website is blocking you, it's probably because of your usage patterns (i.e. you're clogging up their web server by making 1000 requests every minute. That's rude. Don't do that!)

一个例外是如果您在HtmlUnitDriver的无头"模式下使用Selenium.该网站可以进行检测.

One exception would be if you're using Selenium in "headless" mode with the HtmlUnitDriver. The website can detect that.

这篇关于一些网站阻止了硒webdriver，这是如何工作的?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！