告诉 urllib2 使用自定义 DNS [英] Tell urllib2 to use custom DNS

问题描述

我想告诉 urllib2.urlopen(或自定义开启器)使用 127.0.0.1(或 ::1) 来解析地址.但是，我不会更改我的 /etc/resolv.conf.

I'd like to tell urllib2.urlopen (or a custom opener) to use 127.0.0.1 (or ::1) to resolve addresses. I wouldn't change my /etc/resolv.conf, however.

一种可能的解决方案是使用dnspython 之类的工具来查询地址，使用httplib 之类的工具来构建自定义网址开启器.不过，我更愿意告诉 urlopen 使用自定义名称服务器.有什么建议吗?

One possible solution is to use a tool like dnspython to query addresses and httplib to build a custom url opener. I'd prefer telling urlopen to use a custom nameserver though. Any suggestions?

推荐答案

看起来名称解析最终是由 socket.create_connection 处理的.

Looks like name resolution is ultimately handled by socket.create_connection.

-> urllib2.urlopen
-> httplib.HTTPConnection
-> socket.create_connection

虽然一旦设置了Host:"标头，您就可以解析主机并将 IP 地址向下传递到开启程序.

Though once the "Host:" header has been set, you can resolve the host and pass on the IP address through down to the opener.

我建议您将httplib.HTTPConnection 子类化，并在将connect 方法包装起来以修改self.host，然后再将其传递给<代码>socket.create_connection.

I'd suggest that you subclass httplib.HTTPConnection, and wrap the connect method to modify self.host before passing it to socket.create_connection.

然后将HTTPHandler(和HTTPSHandler)子类化，用传递你的HTTPConnection的方法替换http_open方法httplib 自己的do_open.

Then subclass HTTPHandler (and HTTPSHandler) to replace the http_open method with one that passes your HTTPConnection instead of httplib's own to do_open.

像这样:

import urllib2
import httplib
import socket

def MyResolver(host):
  if host == 'news.bbc.co.uk':
    return '66.102.9.104' # Google IP
  else:
    return host

class MyHTTPConnection(httplib.HTTPConnection):
  def connect(self):
    self.sock = socket.create_connection((MyResolver(self.host),self.port),self.timeout)
class MyHTTPSConnection(httplib.HTTPSConnection):
  def connect(self):
    sock = socket.create_connection((MyResolver(self.host), self.port), self.timeout)
    self.sock = ssl.wrap_socket(sock, self.key_file, self.cert_file)

class MyHTTPHandler(urllib2.HTTPHandler):
  def http_open(self,req):
    return self.do_open(MyHTTPConnection,req)

class MyHTTPSHandler(urllib2.HTTPSHandler):
  def https_open(self,req):
    return self.do_open(MyHTTPSConnection,req)

opener = urllib2.build_opener(MyHTTPHandler,MyHTTPSHandler)
urllib2.install_opener(opener)

f = urllib2.urlopen('http://news.bbc.co.uk')
data = f.read()
from lxml import etree
doc = etree.HTML(data)

>>> print doc.xpath('//title/text()')
['Google']

如果你使用HTTPS，显然存在证书问题，你需要填写MyResolver...

Obviously there are certificate issues if you use the HTTPS, and you'll need to fill out MyResolver...

这篇关于告诉 urllib2 使用自定义 DNS的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！