如何在Python中的单元测试场景中模拟HTTP请求 [英] How to Mock an HTTP request in a unit testing scenario in Python

问题描述

我想为我所有与HTTP相关的测试都包含一个Web服务器。它不需要非常复杂。我宁愿不依赖在线。所以我可以测试一下我的程序的一些选项。

I would like to include a Web server for all my test related to HTTP. It doesn't need to be very sophisticated. I would prefer not to be dependent of being online. So I could test some options of my program.

1. 启动服务器


2. 使用适当的mime类型创建一些资源（URI），响应代码等等。


3. 运行测试（最好不要为每个测试启动服务器）。


4. 关闭服务器。

1. Start the server
2. Create a few resources (URI) with appropriate mime types, response code, etc.
3. Run the tests (would be good to not have to start the server for each tests too)
4. Shut down the server.

此代码的任何提示都会有所帮助。我用BaseHTTPServer尝试了一些但尚未成功的东西。 nosetests命令似乎无限期等待。

Any hints on this code would be helpful. I tried a few things with BaseHTTPServer but not successful yet. nosetests command seems to wait indefinitely.

import unittest
from foo import core

class HttpRequests(unittest.TestCase):
    """Tests for HTTP"""

    def setUp(self):
        "Starting a Web server"
        self.port = 8080
        # Here we need to start the server
        #
        # Then define a couple of URIs and their HTTP headers
        # so we can test the code.
        pass

    def testRequestStyle(self):
        "Check if we receive a text/css content-type"
        myreq = core.httpCheck()
        myuri = 'http://127.0.0.1/style/foo'
        myua = "Foobar/1.1"
        self.asserEqual(myreq.mimetype(myuri, myua), "text/css")

    def testRequestLocation(self):
        "another test" 
        pass

    def tearDown(self):
        "Shutting down the Web server"
        # here we need to shut down the server
        pass

谢谢任何帮助。

更新 - 2012：07：10T02：34：00Z

这是给定网站将返回CSS列表的代码。我想测试它是否返回正确的CSS列表。

This is a code which for a given Web site will return the list of CSS. I want to test if it returns the right list of CSS.

import unittest
from foo import core

class CssTests(unittest.TestCase):
    """Tests for CSS requests"""

    def setUp(self):
        self.css = core.Css()
        self.req = core.HttpRequests()

    def testCssList(self):
        "For a given Web site, check if we get the right list of linked stylesheets"
        WebSiteUri = 'http://www.opera.com/'
        cssUriList = [
        'http://www.opera.com/css/handheld.css',
        'http://www.opera.com/css/screen.css',
        'http://www.opera.com/css/print.css',
        'http://www.opera.com/css/pages/home.css']
        content = self.req.getContent(WebSiteUri)
        cssUriListReq = self.css.getCssUriList(content, WebSiteUri)
        # we need to compare ordered list.
        cssUriListReq.sort()
        cssUriList.sort()
        self.assertListEqual(cssUriListReq, cssUriList)

然后在 foo / core.py

import urlparse
import requests
from lxml import etree
import cssutils

class Css:
    """Grabing All CSS for one given URI"""


    def getCssUriList(self, htmltext, uri):
        """Given an htmltext, get the list of linked CSS"""
        tree = etree.HTML(htmltext)
        sheets = tree.xpath('//link[@rel="stylesheet"]/@href')
        for i, sheet in enumerate(sheets):
            cssurl = urlparse.urljoin(uri, sheet)
            sheets[i] = cssurl
        return sheets

现在，代码依赖于在线服务器。它不应该。我希望能够添加大量不同类型的样式表组合并测试协议，然后在解析，组合等方面添加一些选项。

Right now, the code depends on an online server. It should not. I want to be able to add plenty of different types of combination of stylesheets and to test the protocol and then later on some options on their parsing, combinations, etc.

推荐答案

启动Web服务器进行单元测试绝对不是一个好习惯。单元测试应该是简单和孤立的，这意味着他们应该避免执行IO操作。例如。

Starting a web server for unit testing is definitely not a good practice. Unit tests should be simple and isolated, which means that they should avoid performing IO operations for example.

如果你想写的是真正的单元测试那么你应该制作你的自己的测试输入，并查看模拟对象。 Python是一种动态语言，模拟和猴子路径是编写单元测试的简单而强大的工具。特别是，看看优秀的模拟模块。

If what you want to write are really unit tests then you should craft your own test inputs and also look into mock objects. Python being a dynamic language, mocking and monkey pathing are easy and powerful tools for writing unit test. In particular, have a look at the excellent Mock module.

因此，如果我们看一下你的 CssTests 示例，那么你正在尝试测试 css.getCssUriList 能够提取您提供的HTML片段中引用的所有CSS样式表。您在此特定单元测试中所做的不是测试您是否可以发送请求并从网站获得响应，对吧？您只是想确保给定一些HTML，您的函数会返回正确的CSS URL列表。因此，在此测试中，您显然不需要与真实的HTTP服务器通信。

So, if we have a look at your CssTests example, you are trying to test that css.getCssUriList is able to extract all the CSS stylesheet referenced in a piece of HTML you give it. What you are doing in this particular unit test is not testing that you can send a request and get a response from a website, right? You simply want to make sure that given some HTML, your function returns the correct list of CSS URLs. So, in this test, you clearly do not need to talk to a real HTTP server.

我会执行以下操作：

import unittest

class CssListTestCase(unittest.TestCase):

    def setUp(self):
        self.css = core.Css()

    def test_css_list_should_return_css_url_list_from_html(self):
        # Setup your test
        sample_html = """
        <html>
            <head>
                <title>Some web page</title>
                <link rel='stylesheet' type='text/css' media='screen'
                      href='http://example.com/styles/full_url_style.css' />
                <link rel='stylesheet' type='text/css' media='screen'
                      href='/styles/relative_url_style.css' />
            </head>
            <body><div>This is a div</div></body>
        </html>
        """
        base_url = "http://example.com/"

        # Exercise your System Under Test (SUT)
        css_urls = self.css.get_css_uri_list(sample_html, base_url)

        # Verify the output
        expected_urls = [
            "http://example.com/styles/full_url_style.css",
            "http://example.com/styles/relative_url_style.css"
        ]
        self.assertListEqual(expected_urls, css_urls)    

使用依赖注入进行模拟

现在，不太明显的是单元测试 getContent（）你的 core.HttpRequests 类的方法。我想你使用的是HTTP库而不是在TCP套接字上发出自己的请求。

Mocking with Dependency Injection

Now, something less obvious would be unit testing the getContent() method of your core.HttpRequests class. I suppose you are using an HTTP library and not making your own requests on top of TCP sockets.

为了让您的测试保持在单元级别，您不希望通过网络发送任何内容。您可以做些什么来避免这种情况，进行测试确保您使用HTTP lib正确地说。这是关于测试代码的行为而不是它与周围其他对象交互的方式。

To keep your tests at the unit level, you don't want to send anything over the wire. What you can do to avoid that, is having tests that ensure that you make use of your HTTP library correctly. This is about testing not the behaviour of your code but rather the way it interacts with the other objects around it.

这样做的一种方法是依赖于那个库显式：我们可以在 HttpRequests .__ init __ 中添加一个参数来传递一个库的HTTP客户端实例。假设我使用提供 HttpClient 对象的HTTP库，我们可以在该对象上调用 get（）。你可以这样做：

One way to do so would be to make the dependency on that library explicit: we can add a parameter to the HttpRequests.__init__ to pass it an instance of library's HTTP client. Say I use an HTTP library that provides a HttpClient object on which we can call get(). You could do something like:

class HttpRequests(object):

    def __init__(self, http_client):
        self.http_client = http_client

   def get_content(self, url):
        # You could imagine doing more complicated stuff here, like checking the
        # response code, or wrapping your library exceptions or whatever
        return self.http_client.get(url)

我们已经明确了依赖项，现在需要由 HttpRequests 的调用者来满足这个要求：这称为依赖注入（DI）。

We have made the dependency explicit and the requirement now needs to be met by the caller of HttpRequests: this is called Dependency Injection (DI).

DI对两件事非常有用：

DI is very useful for two things:

1. 它避免了你的代码依赖于某些对象的惊喜存在于某个地方


2. 它允许编写测试，根据测试的目标注入不同类型的对象

在这里，我们可以使用我们将给 core.HttpRequests 并且它将在不知不觉中使用，就好像它是真正的库一样。之后，我们可以测试交互是否按预期进行。

Here, we can use a mock object that we will give to core.HttpRequests and that it will use, unknowingly, as if it were the real library. After that, we can test that the interaction was conducted as expected.

import core

class HttpRequestsTestCase(unittest.TestCase):

    def test_get_content_should_use_get_properly(self):
        # Setup

        url = "http://example.com"

        # We create an object that is not a real HttpClient but that will have
        # the same interface (see the `spec` argument). This mock object will
        # also have some nice methods and attributes to help us test how it was used.
        mock_http_client = Mock(spec=somehttplib.HttpClient) 

        # Exercise

        http_requests = core.HttpRequests(mock_http_client)
        content = http_requests.get_content(url)

        # Here, the `http_client` attribute of `http_requests` is the mock object we
        # have passed it, so the method that is called is `mock.get()`, and the call
        # stops in the mock framework, without a real HTTP request being sent.

        # Verify

        # We expect our get_content method to have called our http library.
        # Let's check!
        mock_http_client.get.assert_called_with(url)

        # We can find out what our mock object has returned when get() was
        # called on it
        expected_content = mock_http_client.get.return_value
        # Since our get_content returns the same result without modification,
        # we should have received it
        self.assertEqual(content, expected_content)

我们现在测试了我们的 get_content 方法与我们的HTTP库正确交互。我们已经定义了 HttpRequests 对象的边界并对它们进行了测试，这是我们应该在单元测试级别上进行的。该请求现在在该库的手中，并且它肯定不是我们的单元测试套件的角色来测试库是否按预期工作。

We have now tested that our get_content method interacts correctly with our HTTP library. We have defined the boundaries of our HttpRequests object and tested them, and this is as far as we should go at the unit test level. The request is now in the hand of that library and it is certainly not the role of our unit test suite to test that the library works as expected.

现在想象我们决定使用优秀的请求库。它的API更具程序性，它不会提供我们可以从中获取HTTP请求的对象。相反，我们将导入模块并调用其 get 方法。

Now imagine that we decide to use the great requests library. Its API being more procedural, it does not present an object we can grab to make HTTP requests from. Instead, we would import the module and call its get method.

我们的 HttpRequests core.py 中的类会看到如下内容：

Our HttpRequests class in core.py would then look somethings like the following:

import requests

class HttpRequests(object):

    # No more DI in __init__

    def get_content(self, url):
        # We simply delegate the HTTP work to the `requests` module
        return requests.get(url)

不再有DI，所以现在，我们想知道：

No more DI, so now, we are left wondering:

• 我如何阻止网络互动发生了吗？


• 我如何测试我是否正确使用 requests 模块？

• How am I to prevent network interaction from happening?
• How am I to test that I use the requests module properly?

这是您可以使用动态语言提供的另一种奇妙且有争议的机制：猴子补丁。我们将在运行时用我们制作的对象替换 requests 模块，并在我们的测试中使用。

This is where you can use another fantastic, yet controversial, mechanism that dynamic languages offer: monkey patching. We will replace, at runtime, the requests module with an object we craft and can use in our test.

我们的单元测试将类似于：

Our unit test will then look something like:

import core

class HttpRequestsTestCase(unittest.TestCase):

    def setUp(self):
        # We create a mock to replace the `requests` module
        self.mock_requests = Mock()

        # We keep a reference to the current, real, module
        self.old_requests = core.requests

        # We replace the module with our mock
        core.requests = self.mock_requests

    def tearDown(self):
        # It is very important that each unit test be isolated, so we need
        # to be good citizen and clean up after ourselves. This means that
        # we need to put back the correct `requests` module where it was
        core.requests = self.old_requests

    def test_get_content_should_use_get_properly(self):
        # Setup

        url = "http://example.com"

        # Exercise
        http_client = core.HttpRequests()
        content = http_client.get_content(url)

        # Verify

        # We expect our get_content method to have called our http library.
        # Let's check!
        self.mock_requests.get.assert_called_with(url)

        # We can find out what our mock object has returned when get() was
        # called on it
        expected_content = self.mock_requests.get.return_value
        # Since our get_content returns the same result without modification,
        # we should have received
        self.assertEqual(content, expected_content)

为了使这个过程更简洁， mock 模块有一个 patch 照看脚手架的装饰器。然后我们只需要写：

To make this process less verbose, the mock module has a patch decorator that looks after the scaffolding. We then only need to write:

import core

class HttpRequestsTestCase(unittest.TestCase):

    @patch("core.requests")
    def test_get_content_should_use_get_properly(self, mock_requests):
        # Notice the extra param in the test. This is the instance of `Mock` that the
        # decorator has substituted for us and it is populated automatically.

        ...

        # The param is now the object we need to make our assertions against
        expected_content = mock_requests.get.return_value

结论

保持单元测试小，简单非常重要，快速，自足。依赖于另一台服务器运行的单元测试根本不是单元测试。为了解决这个问题，DI是一个很好的做法，模拟对象是一个很好的工具。

Conclusion

It is very important to keep unit test small, simple, fast, and self-contained. A unit test that relies on another server to be running is simply not a unit test. To help with that, DI is a great practice, and mock objects a great tool.

首先，要理解模拟和概念是不容易的。但是如何使用它们。像每个电动工具一样，它们也可以在你手中爆炸，例如让你相信你已经测试了一些东西，而实际上你却没有。确保模拟对象的行为和输入/输出反映现实是至关重要的。

At first, it is not easy to get your head around the concept of mock and how to use them though. Like every power tool, they can also explode in your hands and for example make you believe you have tested something when in reality you have not. Making sure that the behaviour and input/output of mock objects reflects the reality is paramount.

给定我们从未在单元测试级别与真正的HTTP服务器进行交互，编写集成测试非常重要，这将确保我们的应用程序能够与现实生活中将要处理的服务器进行通信。我们可以通过专门为集成测试设置的完全成熟的服务器来实现这一目标，或者编写一个人为的。

Given that we have never interacted with a real HTTP server at the unit test level, it is important to write Integration Tests that will make sure our application is able to talk to the sort of servers it will deal with in real life. We could do this with a fully fledged server set up specially for Integration Testing, or write a contrived one.

这篇关于如何在Python中的单元测试场景中模拟HTTP请求的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！