如何使用硒捕获网络流量 [英] How to capture network traffic with selenium

问题描述

我正在启动一个新的Django项目，我正在尝试使用Selenium捕获网络流量.

I'm starting a new Django project, I'm trying to capture the network traffic with Selenium.

我已经使用Selenium-wire(MITM代理)实现了这一目标，但是Django不喜欢使用Selenium-wire(必须使用"--nothreading --noreload"启动服务器，连接错误...).我正在寻找通过现代解决方案实现此目标的方法，例如直接解析firefox的网络devtools或使用firefox插件解析这些工具.我正在使用Firefox Geckodriver进行测试.

I have already achieved this objective with Selenium-wire (MITM Proxy), but Django doesn't like to work with selenium-wire ( must start the server with "--nothreading --noreload", connection bug... ). I'm looking for achieve this with modern solutions, like parsing the network devtools of firefox directly or with a firefox addons. I'm using Firefox Geckodriver for my test.

    for x in range(0, 10):
                profile = profile_firefox()
                options = options_firefox()
                driver = webdriver.Firefox(firefox_profile=profile, options=options, executable_path='/Users/*****/Desktop/selenium-master/headless_browser/geckodriver')
                try:
                        driver.set_window_position(0, 0)
                        driver.set_window_size(randint(1024, 2060), randint(1024, 4100))
                        time.sleep(randint(3,10))
                        driver.get(url)
                        wait = ui.WebDriverWait(driver, 10)
                        time.sleep(randint(8,10))
                        if driver.find_element_by_xpath("//*[@id=\"container\"]/main/div/div/div[1]/div[2]/form/div/div[2]/div[1]/button"):
                                driver.find_element_by_xpath("//*[@id=\"container\"]/main/div/div/div[1]/div[2]/form/div/div[2]/div[1]/button").click()
                                del driver.requests
                                time.sleep(randint(8,10))
                                driver.find_element_by_xpath("//*[@id=\"container\"]/main/div/div/div[1]/div[2]/form/div/div[2]/div[1]/button").click()
                                time.sleep(randint(10,20))
                                for request in driver.requests:
                                        if request.path == "https://api.*********.**/*******/*******":
                                                request_api = request
                                                raw = str(request_api.body)
                                                request_api = raw.split(('b\''))
                                                payload_raw = request_api[1]
                                                payload = payload_raw[:-1]
                                                if payload:
                                                        header = request.headers
                                                        time.sleep(8)
                                                        break

                except:
                        print("Houston on a eu un probleme")
                        firefox_closing(driver)

def profile_firefox():
        profile = FirefoxProfile()
        profile.set_preference('permissions.default.image', 2)
        profile.set_preference('dom.ipc.plugins.enabled.libflashplayer.so', 'false')
        profile.set_preference("general.useragent.override", firefox_init())
        profile.set_preference('network.proxy.type', 1)
        profile.set_preference('network.proxy.socks', 'localhost')
        profile.set_preference('network.proxy.socks_port', 9050)
        profile.set_preference("network.proxy.socks_remote_dns", False)
        profile.set_preference("driver.privatebrowsing.autostart", True)
        profile.update_preferences()
        return profile

使用袜子，HTTP，SSL配置进行测试2:

Test 2 with Socks,HTTP,SSL configuration :

server = Server('/Users/*****/Desktop/selenium-master/browsermob-proxy-2.1.4/bin/browsermob-proxy')
server.start()
proxy = server.create_proxy()
proxy.selenium_proxy()#Dont understand what it does ???
port = int(proxy.port)

profile = FirefoxProfile()
profile.set_preference('permissions.default.image', 2)
profile.set_preference('dom.ipc.plugins.enabled.libflashplayer.so', 'false')
profile.set_preference('general.useragent.override', firefox_init())
profile.set_preference('network.proxy.type', 1)
profile.set_preference('network.proxy.socks', 'localhost')
profile.set_preference('network.proxy.socks_port', 9050)
profile.set_preference('network.proxy.ssl', 'localhost')
profile.set_preference('network.proxy.ssl_port', port)
profile.set_preference('network.proxy.http', 'localhost')
profile.set_preference('network.proxy.http_port', port)
profile.set_preference('network.proxy.socks_remote_dns', False)
profile.set_preference('driver.privatebrowsing.autostart', True)
profile.update_preferences()

似乎Http代理覆盖了袜子的配置...

It seems Http proxy override the socks configuration...

非常感谢您对我的代码或解决方案有任何线索或建议.

Thanks a lot if you have any clue or advice about my code or solutions.

推荐答案

Browsermob是正确的方法.

Browsermob is the right way.

我必须了解browsermob的工作原理.对于Tor，您必须启用这样的HTTPTunnelPort配置.

I must understand how browsermob works and tor too. For Tor you must enable the HTTPTunnelPort configuration like this.

tor --HTTPTunnelPort 8088

并配置browsermob以使用它.

And configure browsermob to use it.

proxy_params = {'httpProxy': 'localhost:8088', 'httpsProxy': 'localhost:8088'}
proxy_b = server.create_proxy(params=proxy_params)

谢谢.

这篇关于如何使用硒捕获网络流量的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！