在多个环境中执行编码的UI测试 [英] Execute Coded UI Tests in multiple environments

问题描述

现在，我的编码UI测试使用其app.config来确定其执行的域，该域与环境具有1-1的关系.为了简化它:

Right now my Coded UI Tests use their app.config to determine the domain they execute in, which has a 1-1 relationship with environment. To simplify it:

• www.test.com
• www.UAT.com
• www.prod.com

在App.config中，我有类似的东西:

and in App.config I have something like:

<configuration>
    <appSettings>
        <add key="EnvironmentURLMod" value ="test"/>

为了在不同的环境中运行测试，我在运行之间手动更改了该值.例如，我打开浏览器是这样的:

and to run the test in a different environment, I manually change the value between runs. For instance the I open the browser like this:

browserWindow.NavigateToUrl(new Uri("http://www."
                + ConfigurationManager.AppSettings.Get("EnvironmentURLMod")
                + ".com"));

很明显，这很不雅致.我想我有一个愿景，我们每次运行都会在新的app.config文件中放一些东西，但作为破坏者，此测试将在大约10个环境中运行，而不是在3个环境中运行，并且它可能运行的环境可能会发生变化.

Clearly this is inelegant. I suppose I had a vision where we'd drop in a new app.config for each run, but as a spoiler this test will be run in ~10 environments, not 3, and which environments it may run may change.

我知道我可以将这些环境URL修改解耦到另一个XML文件，并使测试在环境变量的建议，但这将需要为每个环境创建一个测试代理，修改它们的注册表，并在每个环境上运行测试.如果那是必需的，那么可以确定，但是似乎要使用大量的VM带宽来存储字符串.

I know I could decouple these environment URL modifications to yet another XML file, and make the tests access them sequentially in a data-driven scenario. But even this seems like it's not quite what I need, since if one environment fails then the whole test collapses. I've seen Environment Variables as a suggestion, but this would require creating a test agent for each environment, modifying their registries, and running the tests on each of them. If that's what it takes then sure, but it seems like an enormous amount of VM bandwidth to be used for what's a collection of strings.

在理想的世界中，我想将这些URL mod与测试设置"，MTM环境或内部版本相关联.我想针对每个域执行测试套件并分别报告.

In an ideal world, I would like to tie these URL mods to something like Test Settings, MTM environments, or builds. I want to execute the suite of tests for each domain and report separately.

简而言之，参数化这些测试的最佳方法是什么?有没有一种方法不涉及排队新版本或删除配置文件?数据驱动测试是答案吗?我的解决方案结构是否正确?看来这应该是一种常见的情况，但是我的谷歌搜索并不能完全达到我的目的.

In short, what's the best way to parameterize these tests? Is there a way that doesn't involve queuing new builds, or dropping config files? Is Data Driven Testing the answer? Have I structured my solution incorrectly? This seems like it should be such a common scenario, yet my googling doesn't quite get me there.

任何人和所有帮助表示赞赏.

Any and all help appreciated.

推荐答案

这里的答案是数据驱动的测试，不幸的是，即使有优于大多数"选项，也没有总的灵丹妙药.

The answer here is data driven testing, and unfortunately there's no total silver bullet even if there's a "Better than most" option.

使用任何数据源，您都可以在多个环境(或您可以想到的任何其他变量)中迭代测试，并实质上返回3个不同的测试结果-每个排列或数据行一个. 但是您将必须更新断言以显示当前在哪个环境中执行，因为测试结果仅显示数据行0"或默认情况下类似的内容.如果测试通过，您将不知道成功运行的数据行中的实际内容，除非将这些信息嵌入到操作日志中！我很幸运，因为我只是使用URL mod，所以用例会自动执行此操作，但是其他人可能需要自己做.

Using any data source lets you iterate through a test in multiple environments (or any other variable you can think of) and essentially return 3 different test results - one for each permutation or data row. However you'll have to update your assertions to show which environment you're currently executing in, as the test results only show "Data Row 0" or something similar by default. If the test passes, you'll get no clue as to what's actually in the data row for the successful run, unless you embed this information in the action log! I'm lucky that my use case does this automatically since I'm just using a URL mod, but other people may need to do that on their own.

为了允许即时更改正在测试的环境，我们选择使用TestCase数据源.这具有很大的灵活性-可能比使用数据库或XML更具灵活性-但它也有缺点.像所有数据驱动方案一样，您本质上必须将测试用例ID硬编码"到测试方法上方的装饰器中(因为它被视为属性).我希望我们至少要更改使用的测试用例时，可以将app.config放入构建放置位置，但是看起来相反，我们将不得不在解决方案中进行查找+替换. .

To allow on-the-fly changing of what environments we're testing in, we chose to use a TestCase data source. This has a lot of flexibility - potentially more than using a database or XML for instance - but it comes with its own downsides. Like all data driven scenarios, you have to essentially "Hard Code" the test case ID into the decorator above your test method (Because it's considered a property). I was hoping we could drop an app.config into the build drop location when we wanted to change which test case we used, at least, but it looks like instead we're going to have to do a find + replace across a solution instead.

如果有人知道将测试ID或连接字符串的任何其他部分与代码分离的更好方法，我将在此处给您答案.对于其他任何人，您可以在 MSDN 中找到更多信息.

If anyone knows of a better way to decouple the test ID or any other part of the connection string from the code, I'll give you an answer here. For anyone else, you can find more information on MSDN.

这篇关于在多个环境中执行编码的UI测试的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！