这是使用 Python 3 unittest 测试标准输出的正确方法吗? [英] Is this a proper way to test stdout with Python 3 unittest?

问题描述

假设我有一个提交文件，fileFromStudent.py，其中只有:

Suppose I have a submission file, fileFromStudent.py, and the only thing in it is:

print("hello world")

我想测试标准输出，看看学生是否正确地写出了打印语句.根据我所阅读的内容，我已经能够创建以下代码:

I would like to test the stdout to see if the student has properly written out the print statement. Based on what I've read, I've been able to create the following code:

from io import StringIO
from unittest.mock import patch
import unittest, importlib, sys

class TestStringMethods(unittest.TestCase):
    def setUp(self):
        studentSubmission = 'fileFromStudent'

        ## Stores output from print() in fakeOutput
        with patch('sys.stdout', new=StringIO()) as self.fakeOutput:
            ## Loads submission on first test, reloads on subsequent tests
            if studentSubmission in sys.modules:
                importlib.reload(sys.modules[ studentSubmission ] )
            else:
                importlib.import_module( studentSubmission )

    ## Test Cases
    def test_print_passes(self):
        test_case = "Checking Output Statement - Will Pass"
        self.output = self.fakeOutput.getvalue().strip()
        self.assertEqual(self.output, 'hello world', msg=test_case)

    def test_print_fails(self):
        test_case = "Checking Output Statement - Will Fail"
        self.output = self.fakeOutput.getvalue().strip()
        self.assertEqual(self.output, 'hell world', msg=test_case)

if __name__ == '__main__':
    suite = unittest.TestLoader().loadTestsFromTestCase(TestStringMethods)
    testResult = unittest.TextTestRunner(verbosity=2).run(suite)

上述方法有效，但我是否以正确的方式处理事情?我添加的其中一件事是 import.reload() 调用以使学生的程序重新加载.这是因为在前几周，我会让学生使用 print() 作为他们的最终输出(直到我们进入函数).

The above works, but am I going about things the correct way? One of the things I added in was the import.reload() call to make the student's program reload. This is because for the first few weeks, I will have the students using print() as their final output (until we get into functions).

我知道这看起来模棱两可，或者为什么我要因为它有效而烦恼，但是上面的代码是构建它的正确方法还是我完全遗漏了一些使这一切变得简单的东西?

I know it seems ambiguous, or why I should bother since it works, but is the above code a proper way to have this built or am I completely missing something that makes all this simple?

推荐答案

我花了几个星期的时间来解决这个问题，取得了一定的成功、头痛和谷歌.我没有走 Popen 路线的原因之一是我想捕获学生提交的错误代码是否立即崩溃.信不信由你，介绍课程的前几周就是这样.由于我发现的所有内容都是 2011-2012 年的，所以我想我会发布此内容，以便未来的 Google 员工可以找到它.

I've spent a couple weeks working on this with moderate success, headaches, and Google. One of the reasons I did not go the Popen route was that I wanted to capture if students submitted bad code that instantly crashes. Believe or not, the first few weeks of an Intro course are like that. Since everything I've found was from 2011-2012, I figured I'd post this so future Google'ers can find it.

展开我上面写的内容，让我们假设下一个任务是获取输入并说嗨"

Expanding out what I wrote above, let's assume the next assignment was to get an input and say "Hi"

name = input("What's your name? ")
print("Hi " + name)

现在，我想自动化测试，看看我是否可以输入 "Adam" 并返回 "Hi Adam".为此，我选择使用 StringIO 作为我的标准输入 (sys.stdin = StringIO("Adam")).这使我可以控制文本流的来源和去向.此外，我不想看到学生可能发生的所有错误(sys.stderr = StringIO()).

Now, I want to automate the test to see if I can type in "Adam" and get back "Hi Adam". To do this, I chose to use StringIO as my stdin (sys.stdin = StringIO("Adam")). This allows me to have control of where text streams are coming from and going. In addition, I don't want to see all the errors a student might have happen (sys.stderr = StringIO()).

正如我提到的，我选择使用 importlib 而不是 Popen.我想确保如果学生提交了虚假代码，而不是破坏所有内容，而是让我正在运行的任何测试失败.我对 subprocess 和 py.test 进行了试验，虽然它们可能更好、更干净，但我找不到任何对我有意义的关于如何让它移动的东西正确.

As I mentioned, I chose to use importlib instead of Popen. I wanted to ensure that if the Student submitted bogus code, instead of breaking everything, just fail whatever test I was running. I experimented with subprocess and py.test and while they might be a better, cleaner fit, I couldn't find anything that made sense to me on how to get it moving properly.

以下是我最新版本的测试副本:

Below is a copy of my latest version of the test:

from io import StringIO
from unittest.mock import patch
import unittest, importlib, sys, os
from time import sleep

# setup the environment
backup = sys.stderr

class TestTypingExercise(unittest.TestCase):
    def __init__(self, test_name, filename, inputs):
        super(TestTypingExercise, self).__init__(test_name)
        self.library = filename.split('.')[0]
        self.inputs = inputs

    def setUp(self):
        sys.stdin = StringIO(self.inputs[0])
        try:
            ## Stores output from print() in fakeOutput
            with patch('sys.stdout', new=StringIO()) as self.fakeOutput:
                ## Loads submission on first test, reloads on subsequent tests
                if self.library in sys.modules:
                    importlib.reload(sys.modules[ self.library ] )
                else:
                    importlib.import_module( self.library )
        except Exception as e:
            self.fail("Failed to Load - {0}".format(str(e)))

    ## Test Cases
    def test_code_runs(self):
        test_case = "Checking to See if code can run"
        self.assertTrue(True, msg=test_case)

    def test_says_hello(self):
        test_case = "Checking to See if code said 'Hi Adam'"
        # Regex might be cleaner, but this typically solves most cases
        self.output = self.fakeOutput.getvalue().strip().lower()
        self.assertTrue('hi adam' in self.output, msg=test_case)

if __name__ == '__main__':
    ignore_list = ["grader.py"]

    # Run Through Each Submitted File
    directory = os.listdir('.')
    for filename in sorted(directory):
        if (filename.split('.')[-1] != 'py') or (filename in ignore_list):
            continue
        #print("*"*15, filename, "*"*15)

        # 'Disables' stderr, so I don't have to see all their errors
        sys.stderr = StringIO()     # capture output

        # Run Tests Across Student's Submission
        suite = unittest.TestSuite()
        suite.addTest(TestTypingExercise('test_code_runs', filename, 'Adam'))
        suite.addTest(TestTypingExercise('test_says_hello', filename, 'Adam'))
        results = unittest.TextTestRunner().run(suite)

        # Reset stderr
        out = sys.stderr.getvalue() # release output
        sys.stderr.close()  # close the stream 
        sys.stderr = backup # restore original stderr

        # Display Test Results
        print(filename,"Test Results - ", end='')
        if not results.wasSuccessful():
            print("Failed (test cases that failed):")
            for error in results.failures:
                print('\t',error[1].split('\n')[-2])
        else:
            print("Pass!")
        sleep(0.05)

这是最终结果:

StudentSubmission01.py Test Results - Failed (test cases that failed):
     AssertionError: Failed to Load - EOL while scanning string literal (StudentSubmission01.py, line 23)
     AssertionError: Failed to Load - EOL while scanning string literal (StudentSubmission01.py, line 23)
StudentSubmission02.py Test Results - Pass!
StudentSubmission03.py Test Results - Pass!
StudentSubmission04.py Test Results - Pass!
StudentSubmission05.py Test Results - Pass!
StudentSubmission06.py Test Results - Pass!
StudentSubmission07.py Test Results - Pass!
StudentSubmission08.py Test Results - Pass!
StudentSubmission09.py Test Results - Pass!
StudentSubmission10.py Test Results - Pass!
StudentSubmission11.py Test Results - Pass!
StudentSubmission12.py Test Results - Pass!
StudentSubmission13.py Test Results - Pass!
[Finished in 0.9s]

如果我想测试多个不同的输入，我可能需要移动一些东西，但现在这可行.

I might need to move things around if I want to test multiple different inputs, but for now this works.

这篇关于这是使用 Python 3 unittest 测试标准输出的正确方法吗?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！