原子测试在动态创建的环境中有意义吗？ [英] Do atomic tests make sense in dynamically created environments?

问题描述

我们正在构建一个允许用户在这些数据库（WebApp）中创建自定义数据库和存储数据的产品。

We´re building a product that allows users to create custom databases and store data within those DBs (WebApp).

我们测试前端的问题（coffeescript ）是每个测试都应该是原子的，但这需要设置一个数据库，以查看该数据库中的项目是否可以创建并持续存在，或者查看数据库中的更改如何影响项目。

Our issue for testing of the frontend (coffeescript) is that every test should be atomic but that would require setting up a DB for seeing if an item within that DB can be created and persists or to see how changes in a DB affect items.

基本上，问题是进入项目测试所需的设置代码基本上设置了一个新数据库，因此等于测试设置新数据库的代码。

Essentially, the issue is that the setup code needed to get to the item tests basically sets up a new DB and therefore equals the code that tests setting up a new DB.

有两种方法可供使用：

1）使用每组测试创建和拆除新数据库

1) Create and tear down a new DB with each group of tests

• （+）Sorta Atomic（如果设置数据库失败仍然失败）


• （ - ）Takes大量的时间来执行


• （ - ）大量的代码


• （ - ）无法探索创建的环境


• （ - ）凌乱，一切都失败

• (+) Sorta Atomic (still fails if setting up a DB fails)
• (-) Takes a lot of time to execute
• (-) Tons of surounding code
• (-) No way to explore the created environment
• (-) Messy on errors, everything fails

2）逐步进行设置，作为彼此依赖的单独测试，测试开始时的清理程序

2) Do the setup step by step as seperate tests depending on each other, cleanup routine at beginning of a test

• （+）可以通过UI访问创建的环境（不会自动拆除）


• （+）一步一步测试，减少整体/重复代码


• （ - ）测试依赖于彼此（凌乱）


• （ - ）有些整体凌乱

• (+) The created environment can be accessed via the UI (not automatically torn down)
• (+) Step by step testing, less overall/repetitive code
• (-) Tests depended on each other (messy)
• (-) Somewhat overall messy

我们想知道在这样一个动态的环境中测试应该是原子的黄金法则是否合理？

We´re wondering therefore if the golden rule that tests should be atomic makes sense in such a dynamic environment?

推荐答案

基本上，你所说的是集成测试。这些与单元测试不同。集成测试的示例是自动UI测试或编码UI测试。在我参与过的大多数项目中，我们都进行了两种类型的测试，我强烈建议您在项目中同时使用这两种类型。

Basically, what you are talking about is Integration tests. These are different from Unit Tests. Examples of integration test would be Automated UI tests or Coded UI tests. In most of the projects I've worked on we've had both types of tests and I strongly encourage you to have both types in your project too.

背后的哲学这些测试略有不同。

The philosophy behind both these tests is slightly different.

• 单元测试用于测试功能的孤立位。


• 他们意味着非常快。


• 开发人员应该能够在合理的时间内在他们的机器上运行它们。

这种哲学有各种各样的后果。

There are various consequences of this philosophy.

• 因为单元测试正在测试一些孤立的功能，所以你应该使用模拟和存根来隔离环境的其余部分而只关注微不足道的功能。


• 隔离有助于您在编写这些测试时设计思维。事实上，这就是为什么单元测试需要快速的原因，因为开发人员在设计和重新设计过程中积极且不断地改变代码和单元测试。设置，更改和运行单元测试的开销应该非常低。我应该能够忽略除了我试图解决的问题以外的所有事情，并快速迭代并重申我的设计和测试。这是TDD背后的想法，它声称可以帮助编写好的可测试代码。如果你花了很长时间试图设置一个过于复杂的单元测试，那么你必须开始重新考虑你的设计。


• 快速性意味着你可以将它们作为连续的一部分来运行集成构建。


• 缺点是，由于您正在单独测试每个功能，因此您不知道它们是否将作为一个整体协同工作。每次你编写一个模拟器时，你都隐含地假设系统的其余部分是如何工作的，并且系统的其余部分当前正在工作（即，在部署或运行​​过程中没有其他任何部分被破坏）或修补操作系统等。）

• Because unit test is testing an isolated bit of functionality, you should use mocks and stubs to isolate the rest of the environment and only focus on tiny bits of functionality.
• The isolation helps your "design thinking" while writing these tests. In fact this is the reason why the unit tests are required to be fast, because a developer is actively and constantly changing the code and unit tests as part of the design and redesign process. There should be very low overhead to set up, change and run the unit tests. I should be able to ignore everything other than the problem I am trying to solve and quickly iterate and reiterate my designs and tests. This is the idea behind TDD and its claim to help write good testable code. If you are spending a long time trying to set up an overly complex unit test then you have to start reconsidering your design.
• The fast nature means that you could run these as part of your Continuous Integration build.
• The disadvantage is that because you are testing each functionality in isolation you don't know if they will all work together as a whole. Each time you write a mock, you are implicitly baking in an assumption about how the rest of the system works and that the rest of the system is currently working as it is meant to (i.e nothing else is broken as part of your deployment or running or patching of the OS etc.)

集成测试旨在从端到端测试功能。你尽量不要嘲笑或隔离系统的任何部分。

Integration Tests are meant to test the functionality from end to end. You try NOT to mock out or isolate any part of the system.

这种理念还有各种后果。请注意，集成测试不需要快速。

There are again various consequence of this philosophy. Note that there is no requirement for integration tests to be fast.

• 集成测试本质上需要在完全部署后运行（而不是单元测试，可以在你的代码编译）。


• 因为它们需要更长时间，所以不要将它们作为CI环境的一部分运行，但您仍需要定期运行它们。我们通常将它们作为我们夜间构建的一部分运行。或者你可以每天运行两次等。


• 因为集成测试对整个系统采用了黑盒方法，所以它并没有真正帮助你设计思维 关于如何实际构建系统。但它确实有助于您思考整个系统的规范。即 系统应该做什么，而不是 它应该做什么。

• Integration tests, by their very nature need to run after your full deployment (as opposed to unit tests which can be run as soon as your code compiles).
• Because they take longer, you don't run them as part of your CI environment, but you still need to run them regularly. We usually run them as part of our nightly builds. Or you can run it twice daily etc.
• Because the integration tests take a black box approach to the whole system, it doesn't really help you with you "design thinking" about how to actually build the system. But it does help your thinking about the specifications of the system as a whole. i.e What the system should do, not how it should do something.

请注意，在这两种情况下，测试原则仍然适用。每项测试都与其他测试不同。这样，当测试失败时，您可以确定导致其失败的所有条件，并专注于仅修复它。只是集成测试会触及尽可能多的系统部件。

Note that in both cases the rule of tests being atomic still applies. Each test is different from other tests. This way when a test fails you can be sure about all the conditions that are causing it to fail and concentrate on only fixing that. It's just that an integration test touches as many parts your system as possible.

为您举例说明我们当前的项目。

To give you an example on our current project.

让我们说我们需要编写一些功能，要求我们向数据库中添加一个新表，并将其引入所有层以在UI中显示它。

Lets say we need to write a bit of functionality that requires us to add a new table to the DB and bring it through all the layers to show it in the UI.

我们首先创建业务逻辑类，域类，编写适当的Web服务，构建视图模型，修改数据库等。在执行这些操作时，我们编写单元测试来测试我们当前编写的代码。因此，在构建业务逻辑类时，我们会模拟其他所有内容以确保类中的逻辑有效（例如，60岁以上的客户可以获得50％的汽车保险折扣等。）

We start by creating our business logic classes, domain classes, write the appropriate web service, build view models, modify the database etc. While doing each of these we write unit tests to test the code we are currently writing. So when building the business logic classes, we mock out everything else to ensure that the logic in the class is valid (for example, clients over 60 years old get a 50% discount on their car insurance etc.)

一旦我们这样做，我们现在需要更新我们的部署脚本/包等以便能够部署它。即更新数据库创建SQL脚本和数据库更改SQL脚本等（在您的情况下，这将是复杂的过程）。

Once we do that, we now need to update our deployment scripts / packages etc. to be able to deploy it. i.e update the database creation SQL scripts and the database alteration SQL scripts etc. (In your case this will be complex process).

现在我们编写集成测试。在这种情况下，我们可以测试从SQL Server到Web服务。有一个SQL Integration测试基类，它包含每个测试的设置和拆除方法。在设置中，我们使用sql部署脚本创建一个全新的数据库。每个测试还指定一个测试数据sql脚本。因此，例如，此测试数据脚本可能会将新记录插入到年龄为70年的客户端表中。我们将此脚本作为测试安排的一部分运行。然后进行Web服务调用以搜索60岁以上的客户端。这是测试的Act部分，从结果中，我们检查以确保只返回我们插入到DB中的用户。在测试结束时，数据库将被删除。当SQL数据库中的列不可为空或者日期时间列溢出时，我们已经发现了错误，因为.Net中的默认最小日期时间与SQL服务器的最小日期时间不同。

Now we write integration tests. In this case we might test from SQL Server to Web Service. There is a SQL Integration test base class which contains the set up and tear down method for each test. In the set up we create a brand new database using our sql deployment scripts. Each test also specifies a test data sql script. So for example this test data script might insert a new record into the client table whose age is 70 years. We run this script as part of the "Arrange" of our test. Then make a web service call to search for clients older than 60. This is the "Act" part of the test and from the result, we check to make sure that we only get back the user we've inserted into the DB. At the end of the test, the database is deleted. We've caught bugs here when the columns in SQL database aren't nullable or the datetime columns overflow because the default minimum datetime in .Net is a different size from SQL server's minimum datetime.

某些功能要求我们与Oracle数据库进行交互。例如，如果将新记录添加到Oracle，则触发器/数据库过程启动并将该记录传输到SQL，然后我们需要将其启动到层。在这种情况下，我们有一个OracleSQL集成测试基类。正如您可能已经猜到的那样，这遵循一个简单的模式，但是创建了Oracle和SQL dbs将测试数据插入到Oracle中并在测试结束时将它们两个都吹掉。

Some functionality requires us to interact with an Oracle database. For example, if a new record is added to Oracle, then a trigger/db procedure kicks off and transfers that record to SQL and then we need to bring it up the layers. In this case we have an OracleSQL integration test base class. As you might have guessed, this follows a simliar pattern, but creates both Oracle and SQL dbs inserts test data into Oracle and blows them both away at the end of the test.

开发人员通常选择Web服务层来编写集成测试。另一方面，测试人员使用UI自动化工具来确保数据实际显示在屏幕上。例如，他们将记录一个进入网页的测试，点击搜索按钮，将60放入年龄框，点击搜索按钮等。该测试可能利用相同的测试数据sql脚本插入开发人员写的测试数据（或者测试团队可能会来开发人员并请求帮助制作sql脚本以插入他们可以想到的任何高度复杂的数据）。但问题是，一旦创建了测试数据插入脚本，它就会利用相同的底层系统来吹走整个数据库，创建一个新数据库，插入测试数据，并运行指定的测试。

The developers usually pick the Web service layer for writing their integration tests. The testers on the other hand use UI automation tools to make sure that the data is actually showing up on screen. For example they will record a test that goes to web page, clicks search button, puts "60" into the age box, clicks the search button etc. That test might leverages the same test data sql script that inserts test data that the developer wrote (or the testing team might come to the developer and ask help crafting sql scripts to insert whatever highly convoluted data they can think of). But the point is, once the test data insertion script is created, it leverages the same underlying system to blow away the whole db, create a new one, insert test data, and run the specified test.

因此，要回答您的问题，您将需要两种类型的测试，单元测试和集成测试。您可能需要投入一些初始工作来创建一些基类或辅助方法来创建/删除数据库，自动部署以安装/卸载系统的其他组件等。无论如何，您必须为最终部署执行此操作。集成测试还将与您的部署策略密切相关并依赖于您的部署策略。在我看来，这是一个优势，而不是一个缺点。虽然最初设置它可能很痛苦，但集成测试隐式测试的一个问题是您的部署机制。如果在部署/安装系统所需的任何组件时出现任何问题，您希望尽快了解它。不是在你应该部署到生产的前一天。

So, to answer your question, you will need two types of tests, unit tests and integration tests. You might have to put in some initial work into creating some base classes or helper methods to create/delete databases, automating your deployment to install/uninstall other components of your system etc. You will have to do this for your final deployment anyway. Integration tests will also be closely related to and dependent on your deployment strategy. This is an advantage and not a disadvantage in my opinion. While it might be painful at first to set it all up, one of the things your integration tests are implicitly testing is your deployment mechanism. If there are any issues with deploying/installing any of the components required by your system, you want to know about it as quickly as possible. Not the day before you are supposed to be deploying to production.

一套好的测试是非常宝贵的。它还需要孤立，严谨和全面。测试不应该在他们不需要时失败，但更重要的是，他们应该在需要时失败。当它们失败时，您希望它们提供尽可能多的信息并指出确切的故障位置。这使得修复问题变得更加容易。任何时候你建立这个测试套件都会在很短的时间内收回成本。

A good suite of tests is invaluable. It also needs to be isolated, rigorous and comprehensive. The tests shouldn't fail when they don't need to but more importantly, they should fail when they need to. And when they do fail, you want them to provide as much information as possible and point you at the exact location of failure. This makes fixing the issue a much easier task. Any time you put into building this test suite will more than pay for itself in no time.

这篇关于原子测试在动态创建的环境中有意义吗？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！