如何在 Python 的单元测试场景中模拟 HTTP 请求 [英] How to Mock an HTTP request in a unit testing scenario in Python

问题描述

我想为所有与 HTTP 相关的测试包括一个 Web 服务器.它不需要非常复杂.我宁愿不依赖于在线.所以我可以测试我的程序的一些选项.

I would like to include a Web server for all my test related to HTTP. It doesn't need to be very sophisticated. I would prefer not to be dependent of being online. So I could test some options of my program.

1. 启动服务器
2. 使用适当的 mime 类型、响应代码等创建一些资源 (URI).
3. 运行测试(最好不必为每个测试启动服务器)
4. 关闭服务器.

有关此代码的任何提示都会有所帮助.我用 BaseHTTPServer 尝试了一些东西，但还没有成功.nosetests 命令似乎无限期地等待.

Any hints on this code would be helpful. I tried a few things with BaseHTTPServer but not successful yet. nosetests command seems to wait indefinitely.

import unittest
from foo import core

class HttpRequests(unittest.TestCase):
    """Tests for HTTP"""

    def setUp(self):
        "Starting a Web server"
        self.port = 8080
        # Here we need to start the server
        #
        # Then define a couple of URIs and their HTTP headers
        # so we can test the code.
        pass

    def testRequestStyle(self):
        "Check if we receive a text/css content-type"
        myreq = core.httpCheck()
        myuri = 'http://127.0.0.1/style/foo'
        myua = "Foobar/1.1"
        self.asserEqual(myreq.mimetype(myuri, myua), "text/css")

    def testRequestLocation(self):
        "another test" 
        pass

    def tearDown(self):
        "Shutting down the Web server"
        # here we need to shut down the server
        pass

感谢您的帮助.

更新 - 2012:07:10T02:34:00Z

这是一个代码，对于给定的网站将返回 CSS 列表.我想测试它是否返回正确的 CSS 列表.

This is a code which for a given Web site will return the list of CSS. I want to test if it returns the right list of CSS.

import unittest
from foo import core

class CssTests(unittest.TestCase):
    """Tests for CSS requests"""

    def setUp(self):
        self.css = core.Css()
        self.req = core.HttpRequests()

    def testCssList(self):
        "For a given Web site, check if we get the right list of linked stylesheets"
        WebSiteUri = 'http://www.opera.com/'
        cssUriList = [
        'http://www.opera.com/css/handheld.css',
        'http://www.opera.com/css/screen.css',
        'http://www.opera.com/css/print.css',
        'http://www.opera.com/css/pages/home.css']
        content = self.req.getContent(WebSiteUri)
        cssUriListReq = self.css.getCssUriList(content, WebSiteUri)
        # we need to compare ordered list.
        cssUriListReq.sort()
        cssUriList.sort()
        self.assertListEqual(cssUriListReq, cssUriList)

然后在 foo/core.py

import urlparse
import requests
from lxml import etree
import cssutils

class Css:
    """Grabing All CSS for one given URI"""


    def getCssUriList(self, htmltext, uri):
        """Given an htmltext, get the list of linked CSS"""
        tree = etree.HTML(htmltext)
        sheets = tree.xpath('//link[@rel="stylesheet"]/@href')
        for i, sheet in enumerate(sheets):
            cssurl = urlparse.urljoin(uri, sheet)
            sheets[i] = cssurl
        return sheets

目前，代码依赖于在线服务器.它不应该.我希望能够添加大量不同类型的样式表组合并测试协议，然后在它们的解析、组合等方面进行一些选项.

Right now, the code depends on an online server. It should not. I want to be able to add plenty of different types of combination of stylesheets and to test the protocol and then later on some options on their parsing, combinations, etc.

推荐答案

为单元测试启动 Web 服务器绝对不是一个好习惯.单元测试应该简单且隔离，这意味着它们应该避免执行例如 IO 操作.

Starting a web server for unit testing is definitely not a good practice. Unit tests should be simple and isolated, which means that they should avoid performing IO operations for example.

如果您想编写真正的单元测试，那么您应该制作自己的测试输入并查看模拟对象.Python 作为一种动态语言，模拟和猴子路径是编写单元测试的简单而强大的工具.特别是，看看优秀的 Mock 模块.

If what you want to write are really unit tests then you should craft your own test inputs and also look into mock objects. Python being a dynamic language, mocking and monkey pathing are easy and powerful tools for writing unit test. In particular, have a look at the excellent Mock module.

因此，如果我们查看您的 CssTests 示例，您正在尝试测试 css.getCssUriList 是否能够提取片段中引用的所有 CSS 样式表你给它的HTML.您在这个特定的单元测试中所做的并不是测试您可以发送请求并从网站获得响应，对吗?您只是想确保给定一些 HTML，您的函数返回正确的 CSS URL 列表.因此，在这个测试中，您显然不需要与真正的 HTTP 服务器通信.

So, if we have a look at your CssTests example, you are trying to test that css.getCssUriList is able to extract all the CSS stylesheet referenced in a piece of HTML you give it. What you are doing in this particular unit test is not testing that you can send a request and get a response from a website, right? You simply want to make sure that given some HTML, your function returns the correct list of CSS URLs. So, in this test, you clearly do not need to talk to a real HTTP server.

我会做如下的事情:

import unittest

class CssListTestCase(unittest.TestCase):

    def setUp(self):
        self.css = core.Css()

    def test_css_list_should_return_css_url_list_from_html(self):
        # Setup your test
        sample_html = """
        <html>
            <head>
                <title>Some web page</title>
                <link rel='stylesheet' type='text/css' media='screen'
                      href='http://example.com/styles/full_url_style.css' />
                <link rel='stylesheet' type='text/css' media='screen'
                      href='/styles/relative_url_style.css' />
            </head>
            <body><div>This is a div</div></body>
        </html>
        """
        base_url = "http://example.com/"

        # Exercise your System Under Test (SUT)
        css_urls = self.css.get_css_uri_list(sample_html, base_url)

        # Verify the output
        expected_urls = [
            "http://example.com/styles/full_url_style.css",
            "http://example.com/styles/relative_url_style.css"
        ]
        self.assertListEqual(expected_urls, css_urls)    

依赖注入模拟

现在，不太明显的事情是对 core.HttpRequests 类的 getContent() 方法进行单元测试.我想您正在使用 HTTP 库，而不是在 TCP 套接字上发出自己的请求.

Mocking with Dependency Injection

Now, something less obvious would be unit testing the getContent() method of your core.HttpRequests class. I suppose you are using an HTTP library and not making your own requests on top of TCP sockets.

为了使您的测试保持在 unit 级别，您不希望通过网络发送任何内容.您可以做些什么来避免这种情况，即进行测试以确保您正确使用 HTTP 库.这不是测试代码的行为，而是测试它与周围其他对象的交互方式.

To keep your tests at the unit level, you don't want to send anything over the wire. What you can do to avoid that, is having tests that ensure that you make use of your HTTP library correctly. This is about testing not the behaviour of your code but rather the way it interacts with the other objects around it.

这样做的一种方法是明确对该库的依赖:我们可以向 HttpRequests.__init__ 添加一个参数，以将一个库的 HTTP 客户端实例传递给它.假设我使用了一个提供 HttpClient 对象的 HTTP 库，我们可以在该对象上调用 get().你可以这样做:

One way to do so would be to make the dependency on that library explicit: we can add a parameter to the HttpRequests.__init__ to pass it an instance of library's HTTP client. Say I use an HTTP library that provides a HttpClient object on which we can call get(). You could do something like:

class HttpRequests(object):

    def __init__(self, http_client):
        self.http_client = http_client

   def get_content(self, url):
        # You could imagine doing more complicated stuff here, like checking the
        # response code, or wrapping your library exceptions or whatever
        return self.http_client.get(url)

我们已经明确了依赖关系，现在需要 HttpRequests 的调用者来满足要求:这称为依赖注入 (DI).

We have made the dependency explicit and the requirement now needs to be met by the caller of HttpRequests: this is called Dependency Injection (DI).

DI 在两件事上非常有用:

DI is very useful for two things:

1. 它避免了您的代码秘密依赖某个对象存在于某处的意外情况
2. 它允许编写测试，根据测试的目标注入不同类型的对象

在这里，我们可以使用我们将提供给 core.HttpRequests 的模拟对象，并且它会在不知不觉中使用它，就好像它是真正的库一样.之后，我们可以测试交互是否按预期进行.

Here, we can use a mock object that we will give to core.HttpRequests and that it will use, unknowingly, as if it were the real library. After that, we can test that the interaction was conducted as expected.

import core

class HttpRequestsTestCase(unittest.TestCase):

    def test_get_content_should_use_get_properly(self):
        # Setup

        url = "http://example.com"

        # We create an object that is not a real HttpClient but that will have
        # the same interface (see the `spec` argument). This mock object will
        # also have some nice methods and attributes to help us test how it was used.
        mock_http_client = Mock(spec=somehttplib.HttpClient) 

        # Exercise

        http_requests = core.HttpRequests(mock_http_client)
        content = http_requests.get_content(url)

        # Here, the `http_client` attribute of `http_requests` is the mock object we
        # have passed it, so the method that is called is `mock.get()`, and the call
        # stops in the mock framework, without a real HTTP request being sent.

        # Verify

        # We expect our get_content method to have called our http library.
        # Let's check!
        mock_http_client.get.assert_called_with(url)

        # We can find out what our mock object has returned when get() was
        # called on it
        expected_content = mock_http_client.get.return_value
        # Since our get_content returns the same result without modification,
        # we should have received it
        self.assertEqual(content, expected_content)

我们现在已经测试了我们的 get_content 方法与我们的 HTTP 库正确交互.我们已经定义了 HttpRequests 对象的边界并对其进行了测试，这就是我们应该在单元测试级别进行的工作.该请求现在已掌握在该库的手中，我们的单元测试套件当然不会负责测试该库是否按预期工作.

We have now tested that our get_content method interacts correctly with our HTTP library. We have defined the boundaries of our HttpRequests object and tested them, and this is as far as we should go at the unit test level. The request is now in the hand of that library and it is certainly not the role of our unit test suite to test that the library works as expected.

现在假设我们决定使用出色的requests 库.它的 API 更加程序化，它没有提供我们可以抓取来发出 HTTP 请求的对象.相反，我们会导入模块并调用它的 get 方法.

Now imagine that we decide to use the great requests library. Its API being more procedural, it does not present an object we can grab to make HTTP requests from. Instead, we would import the module and call its get method.

我们在 core.py 中的 HttpRequests 类将如下所示:

Our HttpRequests class in core.py would then look somethings like the following:

import requests

class HttpRequests(object):

    # No more DI in __init__

    def get_content(self, url):
        # We simply delegate the HTTP work to the `requests` module
        return requests.get(url)

没有更多的 DI，所以现在，我们想知道:

No more DI, so now, we are left wondering:

• 如何防止网络交互发生?
• 如何测试我是否正确使用了 requests 模块?

您可以在这里使用动态语言提供的另一种奇妙但有争议的机制:猴子补丁.我们将在运行时将 requests 模块替换为我们制作并可以在测试中使用的对象.

This is where you can use another fantastic, yet controversial, mechanism that dynamic languages offer: monkey patching. We will replace, at runtime, the requests module with an object we craft and can use in our test.

我们的单元测试将如下所示:

Our unit test will then look something like:

import core

class HttpRequestsTestCase(unittest.TestCase):

    def setUp(self):
        # We create a mock to replace the `requests` module
        self.mock_requests = Mock()

        # We keep a reference to the current, real, module
        self.old_requests = core.requests

        # We replace the module with our mock
        core.requests = self.mock_requests

    def tearDown(self):
        # It is very important that each unit test be isolated, so we need
        # to be good citizen and clean up after ourselves. This means that
        # we need to put back the correct `requests` module where it was
        core.requests = self.old_requests

    def test_get_content_should_use_get_properly(self):
        # Setup

        url = "http://example.com"

        # Exercise
        http_client = core.HttpRequests()
        content = http_client.get_content(url)

        # Verify

        # We expect our get_content method to have called our http library.
        # Let's check!
        self.mock_requests.get.assert_called_with(url)

        # We can find out what our mock object has returned when get() was
        # called on it
        expected_content = self.mock_requests.get.return_value
        # Since our get_content returns the same result without modification,
        # we should have received
        self.assertEqual(content, expected_content)

为了使这个过程不那么冗长，mock 模块有一个 patch 装饰器来处理脚手架.然后我们只需要写:

To make this process less verbose, the mock module has a patch decorator that looks after the scaffolding. We then only need to write:

import core

class HttpRequestsTestCase(unittest.TestCase):

    @patch("core.requests")
    def test_get_content_should_use_get_properly(self, mock_requests):
        # Notice the extra param in the test. This is the instance of `Mock` that the
        # decorator has substituted for us and it is populated automatically.

        ...

        # The param is now the object we need to make our assertions against
        expected_content = mock_requests.get.return_value

结论

保持单元测试的规模小、简单、快速和独立是非常重要的.依赖另一台服务器运行的单元测试根本不是单元测试.为此，DI 是一种很好的实践，而模拟对象则是一种很好的工具.

Conclusion

It is very important to keep unit test small, simple, fast, and self-contained. A unit test that relies on another server to be running is simply not a unit test. To help with that, DI is a great practice, and mock objects a great tool.

首先，要理解模拟的概念以及如何使用它们并不容易.像每个电动工具一样，它们也可能在您的手中爆炸，例如让您相信您已经测试过某些东西，而实际上您没有.确保模拟对象的行为和输入/输出反映现实至关重要.

At first, it is not easy to get your head around the concept of mock and how to use them though. Like every power tool, they can also explode in your hands and for example make you believe you have tested something when in reality you have not. Making sure that the behaviour and input/output of mock objects reflects the reality is paramount.

鉴于我们从未在单元测试级别与真正的 HTTP 服务器进行过交互，因此编写集成测试以确保我们的应用程序能够与它将在现实生活中处理的那种服务器进行通信非常重要.我们可以使用专门为集成测试设置的成熟服务器来做到这一点，或者编写一个人为的服务器.

Given that we have never interacted with a real HTTP server at the unit test level, it is important to write Integration Tests that will make sure our application is able to talk to the sort of servers it will deal with in real life. We could do this with a fully fledged server set up specially for Integration Testing, or write a contrived one.

这篇关于如何在 Python 的单元测试场景中模拟 HTTP 请求的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！