在多个环境中执行编码的 UI 测试 [英] Execute Coded UI Tests in multiple environments

问题描述

现在我的编码 UI 测试使用他们的 app.config 来确定他们执行的域，它与环境有 1-1 的关系.简化一下:

• www.test.com
• www.UAT.com
• www.prod.com

在 App.config 我有类似的东西:

<预><代码><配置><应用设置><add key="EnvironmentURLMod" value ="test"/>

为了在不同的环境中运行测试，我手动更改了两次运行之间的值.例如我这样打开浏览器:

browserWindow.NavigateToUrl(new Uri("http://www.")+ ConfigurationManager.AppSettings.Get("EnvironmentURLMod")+ ".com"));

显然这是不雅的.我想我有一个愿景，我们会为每次运行添加一个新的 app.config，但作为剧透，这个测试将在大约 10 个环境中运行，而不是 3 个，并且它可能运行的环境可能会改变.

我知道我可以将这些环境 URL 修改解耦到另一个 XML 文件，并使测试在 数据驱动的场景.但即使这似乎也不是我所需要的，因为如果一个环境失败，那么整个测试就会崩溃.我已经看到 环境变量 作为建议，但这需要为每个环境创建一个测试代理，修改它们的注册表，并在每个环境上运行测试.如果这就是它所需要的那么肯定，但似乎有大量的 VM 带宽用于字符串集合.

在理想的世界中，我希望将这些 URL 模块与测试设置、MTM 环境或构建等内容联系起来.我想为每个域执行测试套件并单独报告.

简而言之，参数化这些测试的最佳方法是什么?有没有一种不涉及排队新构建或删除配置文件的方法?数据驱动测试是答案吗?我是否错误地构建了我的解决方案?这似乎应该是一个很常见的场景，但我的谷歌搜索并没有完全让我到达那里.

感谢所有帮助.

解决方案

这里的答案是数据驱动测试，不幸的是，即使有比大多数更好"的选项，也没有万能的灵丹妙药.

使用任何数据源都可以让您在多个环境(或您能想到的任何其他变量)中迭代测试，并基本上返回 3 个不同的测试结果 - 每个排列或数据行一个.但是您必须更新断言以显示您当前正在执行的环境，因为默认情况下测试结果仅显示数据行 0"或类似内容.如果测试通过，您将无法了解成功运行的数据行中实际包含的内容，除非您将此信息嵌入到操作日志中！我很幸运，我的用例会自动执行此操作，因为我只是使用 URL mod，但其他人可能需要自己执行此操作.

为了允许即时更改我们正在测试的环境，我们选择使用 TestCase 数据源.这具有很大的灵活性——例如可能比使用数据库或 XML 更灵活——但它也有其自身的缺点.与所有数据驱动的场景一样，您必须将测试用例 ID 从本质上硬编码"到测试方法上方的装饰器中(因为它被视为一个属性).当我们想要更改我们使用的测试用例时，我希望我们可以将 app.config 放入构建放置位置，至少，但看起来我们将不得不在解决方案中进行查找 + 替换.

如果有人知道将测试 ID 或连接字符串的任何其他部分与代码分离的更好方法，我会在这里给你一个答案.对于其他人，您可以在 MSDN 上找到更多信息.

Right now my Coded UI Tests use their app.config to determine the domain they execute in, which has a 1-1 relationship with environment. To simplify it:

• www.test.com
• www.UAT.com
• www.prod.com

and in App.config I have something like:

<configuration>
    <appSettings>
        <add key="EnvironmentURLMod" value ="test"/>

and to run the test in a different environment, I manually change the value between runs. For instance the I open the browser like this:

browserWindow.NavigateToUrl(new Uri("http://www."
                + ConfigurationManager.AppSettings.Get("EnvironmentURLMod")
                + ".com"));

Clearly this is inelegant. I suppose I had a vision where we'd drop in a new app.config for each run, but as a spoiler this test will be run in ~10 environments, not 3, and which environments it may run may change.

I know I could decouple these environment URL modifications to yet another XML file, and make the tests access them sequentially in a data-driven scenario. But even this seems like it's not quite what I need, since if one environment fails then the whole test collapses. I've seen Environment Variables as a suggestion, but this would require creating a test agent for each environment, modifying their registries, and running the tests on each of them. If that's what it takes then sure, but it seems like an enormous amount of VM bandwidth to be used for what's a collection of strings.

In an ideal world, I would like to tie these URL mods to something like Test Settings, MTM environments, or builds. I want to execute the suite of tests for each domain and report separately.

In short, what's the best way to parameterize these tests? Is there a way that doesn't involve queuing new builds, or dropping config files? Is Data Driven Testing the answer? Have I structured my solution incorrectly? This seems like it should be such a common scenario, yet my googling doesn't quite get me there.

Any and all help appreciated.

解决方案

The answer here is data driven testing, and unfortunately there's no total silver bullet even if there's a "Better than most" option.

Using any data source lets you iterate through a test in multiple environments (or any other variable you can think of) and essentially return 3 different test results - one for each permutation or data row. However you'll have to update your assertions to show which environment you're currently executing in, as the test results only show "Data Row 0" or something similar by default. If the test passes, you'll get no clue as to what's actually in the data row for the successful run, unless you embed this information in the action log! I'm lucky that my use case does this automatically since I'm just using a URL mod, but other people may need to do that on their own.

To allow on-the-fly changing of what environments we're testing in, we chose to use a TestCase data source. This has a lot of flexibility - potentially more than using a database or XML for instance - but it comes with its own downsides. Like all data driven scenarios, you have to essentially "Hard Code" the test case ID into the decorator above your test method (Because it's considered a property). I was hoping we could drop an app.config into the build drop location when we wanted to change which test case we used, at least, but it looks like instead we're going to have to do a find + replace across a solution instead.

If anyone knows of a better way to decouple the test ID or any other part of the connection string from the code, I'll give you an answer here. For anyone else, you can find more information on MSDN.

这篇关于在多个环境中执行编码的 UI 测试的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！