Python中最快的代理迭代 [英] Fastest Proxy Iteration in Python
本文介绍了Python中最快的代理迭代的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!
问题描述
假设我有一个包含10,000多个代理的列表
Let's say I have a list that contains 10,000+ proxies
proxy_list = ['ip:port','ip:port',.....10,000+ items]
我如何迭代它以获取适用于我的PC的代理?使用以下代码可以找到它,但是完成过程需要5 * 10,000秒.我将如何更快地遍历列表?
How do I iterate it to get the proxies that works for my pc? Using the following code it is possible to find it , but takes 5*10,000 seconds to get completed. How would I iterate through the list faster?
import requests
result=[]
for I in proxy_list:
try:
requests.get('http:\\www.httpbin.org\ip',proxies = {'https' : I, 'http' : I } ,timeout = 5)
result.append(I)
except:
pass
推荐答案
您可以使用线程,这将允许程序一次检查多个代理.
You could use threading, this would allow the program to check multiple proxies at once.
import requests
import threading
import concurrent.futures
appendLock = threading.Lock() """This is to keep multiple threads from appending
to the list at the same time"""
workers = 10 """This is the number of threads that will iterate through your proxy list.
In my experience, increasing this number higher than 30 causes problems."""
proxy_list = ['ip:port','ip:port',.....10,000+ items]
result = []
def proxyCheck(proxy):
try:
requests.get('http://www.httpbin.org/ip',proxies = {'https' : I, 'http' : I } ,timeout = 5)
with appendLock:
result.append(I)
except:
pass
with concurrent.futures.ThreadPoolExecutor(max_workers=workers) as executor:
for proxy in proxy_list:
executor.submit(proxyCheck(proxy))
这篇关于Python中最快的代理迭代的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!
查看全文