设置代理隐藏我的IP地址以使用scrapy抓取网页 [英] set proxy To hide my IP address for scraping the webpage using scrapy

问题描述

我正在使用scrapy抓取网站，现在我需要设置代理处理已发送的请求.谁能帮我在scrapy应用程序中解决这个设置代理.如果有，也请提供任何示例链接.我需要解决这个请求来自哪个 IP.

I am using scrapy to crawl website now I need to set proxy handle the request which has been sent. Can anyone help me solve this set proxy in scrapy app. Please give any sample link too if you have so. And I need solution that from which IP this request is going.

推荐答案

你可以通过下面的代码找到此处:

You can do it through the code below found here:

1 – 创建一个名为 middlewares.py 的新文件并将其保存在您的 scrapy 项目中，并向其中添加以下代码.

1 – Create a new file called middlewares.py and save it in your scrapy project and add the following code to it.

# Importing base64 library because we'll need it ONLY
#in case if the proxy we are going to use requires authentication
import base64

# Start your middleware class
class ProxyMiddleware(object):
  # overwrite process request
  def process_request(self, request, spider):
    # Set the location of the proxy
    request.meta['proxy'] = "http://YOUR_PROXY_IP:PORT"

    # Use the following lines if your proxy requires authentication
    proxy_user_pass = "USERNAME:PASSWORD"
    # setup basic authentication for the proxy
    encoded_user_pass = base64.encodestring(proxy_user_pass)
    request.headers['Proxy-Authorization'] = 'Basic ' + encoded_user_pass

2 – 打开您项目的配置文件 (./project_name/settings.py) 并添加以下代码

2 – Open your project’s configuration file (./project_name/settings.py) and add the following code

DOWNLOADER_MIDDLEWARES = {
'scrapy.contrib.downloadermiddleware.httpproxy.HttpProxyMiddleware': 110,
'project_name.middlewares.ProxyMiddleware': 100,
}

此外，您可以在 scrapy 中使用多个代理.更多信息可以可在此处找到.

Also, you can use multiple proxies with scrapy. More information can be found here.

这篇关于设置代理隐藏我的IP地址以使用scrapy抓取网页的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！