为AWS Device Farm构建XCTest UI测试 [英] Building XCTest UI tests for AWS Device Farm
问题描述
我正在尝试设置我们的iOS XCUITests以在AWS Device Farm上运行，但无论我如何构建和上传它们，测试都无法运行。它们在XCode本地运行时执行并传递，但不在AWS Device Farm上运行。
我设置了一个绝对最简单的应用程序，我可以设法尝试隔离这个问题。它由一个带有单个Label的View和一个验证标签存在的UI Test组成。
这是AWS Device Farm界面中出现的错误。
深入查看日志文件会显示此信息（在应用程序输出日志文件）：
2016-04-03 19：07：39.842 XCTRunner [195：28521]正在运行测试...
2016-04-03 19：07：39.868 XCTRunner [195：28521]无法从指定路径加载配置数据;错误：文件名无效。
2016-04-03 19：07：39.872 XCTRunner [195：28521]在/ var / mobile / Containers / Bundle / Application / 883D7F70-E525-4478-9E5E-C87F72A83879 / MyTestAppUITests-Runner中寻找测试包。 app / PlugIns
2016-04-03 19：07：39.875 XCTRunner [195：28521]在/ var / mobile / Containers / Bundle / Application / 883D7F70-E525-4478-9E5E-C87F72A83879 / MyTestAppUITests-找到测试包Runner.app/PlugIns/MyTestAppUITests.xctest
2016-04-03 19：07：39.878 XCTRunner [195：28521]在/ var / mobile / Containers / Bundle / Application / 883D7F70-E525-4478-中寻找配置9E5E-C87F72A83879 / MyTestAppUITests-Runner.app / PlugIns / MyTestAppUITests.xctest
2016-04-03 19：07：39.879 XCTRunner [195：28521]未找到任何配置，创建将运行所有测试的默认配置。
2016-04-03 19：07：39.993 XCTRunner [195：28521] XCTestConfigurationHook：使用Test MyTestAppUITests / testExample
2016-04-03 19：07：39.995 XCTRunner [195：28521] XCTestConfigurationHook：使用测试MyTestAppUITests / testExample
2016-04-03 19：07：39.995 XCTRunner [195：28521] XCTestConfigurationHook：使用Test MyTestAppUITests / testExample
测试套件'选定测试'开始于2016-04-03 19： 07：40.008
测试套件'MyTestAppUITests'开始于2016-04-03 19：07：40.014
测试用例' - [MyTestAppUITests testExample]'开始。
2016-04-03 19：07：40.021 XCTRunner [195：28521] ***断言失败 - [XCUIApplication init]，/ Library / Books / com.apple.xbs / Sources / XCTest_iOS / XCTest-10112 / XCTestFramework / UI Testing / XCUIApplication.m:72
2016-04-03 19：07：40.022 XCTRunner [195：28521] XCTestConfigurationHook：忽略跳过的测试
2016-04-03 19：07：40.023 XCTRunner [195：28521] XCTestConfigurationHook：使用Test MyTestAppUITests / testExample
2016-04-03 19：07：40.025 XCTRunner [195：28521] XCTestConfigurationHook：使用会话ID 75B97C9E-6F14-4F88-A242-64B79C9A0F04
< unknown>：0：错误： - [MyTestAppUITests testExample]：failed：捕获NSInternalInconsistencyException，没有通过测试配置指定的目标应用程序路径：< XCTestConfiguration：0x1652af60>
testBundleURL：file：/// var / mobile / Containers / Bundle / Application / 883D7F70-E525-4478-9E5E-C87F72A83879 / MyTestAppUITests-Runner.app / PlugIns / MyTestAppUITests.xctest /
testBundleRelativePath :( null）
productModuleName：（null）
testsToSkip：（null）
testsToRun：MyTestAppUITests / testExample
reportResultsToIDE：no
sessionIdentifier：< __ NSConcreteUUID 0x1655e5c0> 75B97C9E-6F14-4F88-A242-64B79C9A0F04
pathToXcodeReportingSocket：（null）
disablePerformanceMetrics：no
treatMissingBaselinesAsFailures：no
baselineFileURL：（null）
baselineFileRelativePath：（null）
targetApplicationPath：（null）
targetApplicationBundleID：（null）
reportActivities：no
testsMustRunOnMainThread：no
initializeForUITesting：no
（
0 CoreFoundation 0x22c90123< redacted> + 150
1 libobjc.A.dylib 0x22436e17 objc_exception_throw + 38
2 CoreFoundation 0x22c8ffe1< redacted> + 0
3 Foundation 0x23463b19< redacted> + 92
4 XCTest 0x002625a1 - [XCUIApplication init] + 216
5 MyTestAppUITests 0x01d8a4b7 - [MyTes tAppUITests setUp] + 182
6 XCTest 0x002433bf __24- [XCTestCase invokeTest] _block_invoke_2 + 198
7 XCTest 0x00274661 - [XCTestContext performInScope：] + 308
8 XCTest 0x002432f3 - [XCTestCase invokeTest] + 184
9 XCTest 0x00243a4f - [XCTestCase performTest：] + 566
10 XCTest 0x00241375 - [XCTestSuite performTest：] + 460
11 XCTest 0x00241375 - [XCTestSuite performTest：] + 460
12 XCTest 0x0024e88d - [XCTestObservationCenter _observeTestExecutionForBlock：] + 716
13 XCTest 0x00275c7f _XCTestMain + 1150
14 CoreFoundation 0x22c535b1< redacted> + 12
15 CoreFoundation 0x22c5306d< redacted> + 216
16 CoreFoundation 0x22c515e1< redacted> + 784
17 CoreFoundation 0x22ba4bf9 CFRunLoopRunSpecific + 520
18 CoreFoundation 0x22ba49e5 CFRunLoopRunInMode + 108
19 GraphicsServices 0x23df0ac9 GSEventRunModal + 160
20 UIKit 0x26e34ba1 UIApplicationMain + 144
21 XCTRunner 0x000fa255 XCTRunner + 33365
22 libdyld.dylib 0x22853873< redacted> + 2
）
测试用例' - [MyTestAppUITests testExample]'失败（0.239秒）。
测试套件'MyTestAppUITests'在2016-04-03 19：07：40.256失败了。
执行1次测试，1次失败（1次意外），0.239（0.242）秒
测试套件选定测试在2016-04-03 19：07：40.259失败。
执行1次测试，1次失败（1次意外），0.239（0.251）秒
对我来说，MyAppUITests-Runner.app应用程序看起来没有包含允许它定位正确应用程序的配置文件。所以我怀疑它与我用来构建和上传应用程序及其UI测试的过程有关。
我用来生成Application ipa文件的过程是：
- 在Xcode中，单击产品>存档
- 在管理器窗口中选择存档，单击导出按钮
- 选择保存以进行特别部署并单击下一步
- 选择使用本地签名资产进行配置
- 选择为所有兼容设备导出一个应用
- 包含无线安装的清单已经过UN检查
- 从bitcode重建已选中
- 将ipa导出到桌面上的文件夹
我用来生成UITests ipa文件的过程是（按照
- 选择包含iPad Air 2，iPad Air和iPad 2的设备池，全部运行iOS 9.2.1
- 将设备状态保留为默认设置
- 单击开始运行
请注意，如果我使用相同的应用程序ipa文件并运行内置模糊测试，这些工作没有问题 - 我可以看到我的测试应用程序及其单一标签的截图。
我看不出任何明显的我是什么失踪。我需要做些什么来完成这项工作？
我在AWS Device Farm团队工作。
您在下面引用的分析是在正确的路径上。
对我来说，看起来像MyAppUITests-Runner.app应用程序还没有t包含一个配置文件，允许它定位正确的应用程序。所以我怀疑它与我用来构建和上传应用程序及其UI测试的过程有关。
我们目前修复.ipa中缺少.xctest配置文件时未检测到测试的问题。
此问题的解决方法在修复以使您运行之前是在本地运行测试，这将在.app / Plugins文件夹下生成.xctestconfiguration文件
在打包上传到设备场之前，请确保.c包下的xctest配置文件
Fuzz测试将起作用，因为它不使用您的测试代码。它只需要你的应用程序并对其运行自定义模糊测试。
关于* -Runner.app的位置的另一个注意事项是你应该能够找到它来自Xcode的Products文件夹，但只要你能得到它就应该没事。
更新
这个问题一直没问题自帖子的原始时间起修复，并且不再需要在测试运行器中包含xctestconfiguration文件。
I'm trying to set up our iOS XCUITests to run on AWS Device Farm, but it seems no matter how I build and upload them, the tests will not run. They execute and pass when run locally in XCode, but do not run on AWS Device Farm.
I've set up the absolute simplest app I can manage to try to isolate this issue. It consists of a single View with a single Label, and a UI Test which verifies that the label exists.
Here is the error that appears in AWS Device Farm's interface.
Digging into the log files reveals this (in the "Application Output" log file):
2016-04-03 19:07:39.842 XCTRunner[195:28521] Running tests...
2016-04-03 19:07:39.868 XCTRunner[195:28521] Unable to load configuration data from specified path ; error: The file name is invalid.
2016-04-03 19:07:39.872 XCTRunner[195:28521] Looking for test bundles in /var/mobile/Containers/Bundle/Application/883D7F70-E525-4478-9E5E-C87F72A83879/MyTestAppUITests-Runner.app/PlugIns
2016-04-03 19:07:39.875 XCTRunner[195:28521] Found test bundle at /var/mobile/Containers/Bundle/Application/883D7F70-E525-4478-9E5E-C87F72A83879/MyTestAppUITests-Runner.app/PlugIns/MyTestAppUITests.xctest
2016-04-03 19:07:39.878 XCTRunner[195:28521] Looking for configurations in /var/mobile/Containers/Bundle/Application/883D7F70-E525-4478-9E5E-C87F72A83879/MyTestAppUITests-Runner.app/PlugIns/MyTestAppUITests.xctest
2016-04-03 19:07:39.879 XCTRunner[195:28521] No configurations found, creating a default configuration that will run all tests.
2016-04-03 19:07:39.993 XCTRunner[195:28521] XCTestConfigurationHook: Using Test MyTestAppUITests/testExample
2016-04-03 19:07:39.995 XCTRunner[195:28521] XCTestConfigurationHook: Using Test MyTestAppUITests/testExample
2016-04-03 19:07:39.995 XCTRunner[195:28521] XCTestConfigurationHook: Using Test MyTestAppUITests/testExample
Test Suite 'Selected tests' started at 2016-04-03 19:07:40.008
Test Suite 'MyTestAppUITests' started at 2016-04-03 19:07:40.014
Test Case '-[MyTestAppUITests testExample]' started.
2016-04-03 19:07:40.021 XCTRunner[195:28521] *** Assertion failure in -[XCUIApplication init], /Library/Caches/com.apple.xbs/Sources/XCTest_iOS/XCTest-10112/XCTestFramework/UI Testing/XCUIApplication.m:72
2016-04-03 19:07:40.022 XCTRunner[195:28521] XCTestConfigurationHook: Ignoring skipped tests
2016-04-03 19:07:40.023 XCTRunner[195:28521] XCTestConfigurationHook: Using Test MyTestAppUITests/testExample
2016-04-03 19:07:40.025 XCTRunner[195:28521] XCTestConfigurationHook: Using Session ID 75B97C9E-6F14-4F88-A242-64B79C9A0F04
<unknown>:0: error: -[MyTestAppUITests testExample] : failed: caught "NSInternalInconsistencyException", "No target application path specified via test configuration: <XCTestConfiguration: 0x1652af60>
testBundleURL:file:///var/mobile/Containers/Bundle/Application/883D7F70-E525-4478-9E5E-C87F72A83879/MyTestAppUITests-Runner.app/PlugIns/MyTestAppUITests.xctest/
testBundleRelativePath:(null)
productModuleName:(null)
testsToSkip:(null)
testsToRun:MyTestAppUITests/testExample
reportResultsToIDE:no
sessionIdentifier:<__NSConcreteUUID 0x1655e5c0> 75B97C9E-6F14-4F88-A242-64B79C9A0F04
pathToXcodeReportingSocket:(null)
disablePerformanceMetrics:no
treatMissingBaselinesAsFailures:no
baselineFileURL:(null)
baselineFileRelativePath:(null)
targetApplicationPath:(null)
targetApplicationBundleID:(null)
reportActivities:no
testsMustRunOnMainThread:no
initializeForUITesting:no
"
(
0 CoreFoundation 0x22c90123 <redacted> + 150
1 libobjc.A.dylib 0x22436e17 objc_exception_throw + 38
2 CoreFoundation 0x22c8ffe1 <redacted> + 0
3 Foundation 0x23463b19 <redacted> + 92
4 XCTest 0x002625a1 -[XCUIApplication init] + 216
5 MyTestAppUITests 0x01d8a4b7 -[MyTestAppUITests setUp] + 182
6 XCTest 0x002433bf __24-[XCTestCase invokeTest]_block_invoke_2 + 198
7 XCTest 0x00274661 -[XCTestContext performInScope:] + 308
8 XCTest 0x002432f3 -[XCTestCase invokeTest] + 184
9 XCTest 0x00243a4f -[XCTestCase performTest:] + 566
10 XCTest 0x00241375 -[XCTestSuite performTest:] + 460
11 XCTest 0x00241375 -[XCTestSuite performTest:] + 460
12 XCTest 0x0024e88d -[XCTestObservationCenter _observeTestExecutionForBlock:] + 716
13 XCTest 0x00275c7f _XCTestMain + 1150
14 CoreFoundation 0x22c535b1 <redacted> + 12
15 CoreFoundation 0x22c5306d <redacted> + 216
16 CoreFoundation 0x22c515e1 <redacted> + 784
17 CoreFoundation 0x22ba4bf9 CFRunLoopRunSpecific + 520
18 CoreFoundation 0x22ba49e5 CFRunLoopRunInMode + 108
19 GraphicsServices 0x23df0ac9 GSEventRunModal + 160
20 UIKit 0x26e34ba1 UIApplicationMain + 144
21 XCTRunner 0x000fa255 XCTRunner + 33365
22 libdyld.dylib 0x22853873 <redacted> + 2
)
Test Case '-[MyTestAppUITests testExample]' failed (0.239 seconds).
Test Suite 'MyTestAppUITests' failed at 2016-04-03 19:07:40.256.
Executed 1 test, with 1 failure (1 unexpected) in 0.239 (0.242) seconds
Test Suite 'Selected tests' failed at 2016-04-03 19:07:40.259.
Executed 1 test, with 1 failure (1 unexpected) in 0.239 (0.251) seconds
Which to me looks like the MyAppUITests-Runner.app application hasn't included a configuration file which allows it to target the correct application. So I suspect it's something to do with the process I am using to build and upload the application and its UI Tests.
The process I am using to generate the Application ipa file is:
- In Xcode, click Product > Archive
- Select the Archive in the Organizer window and click the Export button
- Select "Save for Ad Hoc Deployment" and click next
- Select "use local signing assets" for provisioning
- Select "Export one app for all compatible devices"
- "Include manifest for over-the-air installation" is UN-checked
- "Rebuild from bitcode" is checked
- Export the ipa to a folder on desktop
The process I use to generate the UITests ipa file is (following instructions from http://docs.aws.amazon.com/devicefarm/latest/developerguide/test-types-ios-xctest-ui.html):
- In Xcode, Select "Generic iOS Device" as the build target
- Click Product > Build For > Testing
- Open the XCode Derived Data folder for the project, go to Build > Intermediates > Debug-iphoneos. This contains 2 files: MyTestApp.app and MyTestAppUITests-Runner.app
- Create a new directory called "Payload" on desktop
- Copy the MyTestAppUITests-Runner.app file into the Payload directory
- Right click Payload and select "Compress 'Payload'"
- Rename the generated Payload.zip file to mytestapp-uitests.ipa
I have also tried including both MyTestApp.app and MyTestAppUITests-Runner.app into the mytestapp-uitests.ipa file, but this doesn't work either.
To run the tests in Device Farm:
- In my testing project, click "Create a new run"
- Select Android/iOS
- Upload the MyTestApp.ipa file generated earlier (from Exporting the Archive) and click Next
- Select "XCTest UI" from the list, and upload the mytestapp-uitests.ipa file
- Select device pool which consists of iPad Air 2, iPad Air, and iPad 2 all running iOS 9.2.1
- Leave Device State at default settings
- Click Start Run
Note that if I use the same application ipa file and run the built-in Fuzz tests, these work with no problems -- I can see the screenshots with my test app with its single label.
I can't see anything obvious which I'm missing. What do I need to do to make this work?
I work for the AWS Device Farm team. Your analysis quoted below is on the right path.
Which to me looks like the MyAppUITests-Runner.app application hasn't included a configuration file which allows it to target the correct application. So I suspect it's something to do with the process I am using to build and upload the application and its UI Tests.
We are currently fixing an issue where the tests are not picked up if the .xctestconfiguration file is missing from the .ipa.
A workaround for this issue until it is fixed to get you running is to run your tests locally which will generate .xctestconfiguration file under your .app/Plugins folder
Before packaging for upload to device farm just make sure that the xctestconfiguration file is present under the .app bundle
The Fuzz tests will work because it does not use your test code. It takes just takes your app and runs a custom fuzz test against it.
Another note about the location of *-Runner.app is you should be able to find it under the Products folder from Xcode but as long as you can get to it things should be fine.
Update This issue has been fixed since the original time of the post and there is no longer a need to have a xctestconfiguration file in your tests runner.
这篇关于为AWS Device Farm构建XCTest UI测试的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
