KIF:如何自动运行/压力测试 iOS 应用程序以找出罕见 UI 错误的原因? [英] KIF: How to auto-run/stress test an iOS app to find the cause of a rare UI bug?
问题描述
注意:我在标题中添加了 kif 只是为了搜索索引，考虑到大部分答案都是讨论它
我正在为 iOS 寻找类似 selenium 的东西，基本上是一个测试自动化/单元测试框架，它可以多次运行某个 UI 场景直到它崩溃，这将帮助我缩小导致 UI 错误的原因非常罕见且随机地发生.
(顺便说一句，我已经对数据源/表交互的每一行代码进行了 NSLogged 并花费了数小时分析潜在原因……但没有发现任何结论……再次，这个错误很少发生).
我查看了一些 和 KIF.我最终决定使用 KIF 同时借用 cucumber's Gherkin 语法 来描述我的单元测试.
我选择 KIF(而不是 Frank)的原因是 KIF 是 100% 基于 obj-c，而不是使用 ruby 作为Frank 也是如此.所以设置更简单，更适用于我狭隘的测试用例需求.话虽如此，我承认如果我的应用程序更复杂(即使用来自多个服务器的输入等)，Frank 会更有用.你可以看到这个优秀的最后一个季度演示文稿以了解更多关于 KIF、Frank 和其他自动化测试框架的优缺点，包括 Apple 自己的 UI 自动化.
使用 KIF 后，我发现了导致上述错误的错误，并且我可以 100% 使用 KIF 重现它！它很少发生的原因是因为它只有在我非常快地点击屏幕时才会发生..并且由于 KIF 会自动执行这些步骤..它以非常快的速度执行它们..这暴露了错误:).
以下是我用于测试的代码示例.这只是为了让您快速了解 KIF(和 Gherkin)可以为您做什么:
在一个文件中我指定了我想要运行的场景:
- (void)initializeScenarios;{[self addScenario:[KIFTestScenario scenarioToCompleteSignInAndLoadInbox]];[self addScenario:[KIFTestScenario scenarioToFillAttachmentsWithData]];[self addScenario:[KIFTestScenario 场景ToViewAndLoadFileBucket]];[self addScenario:[KIFTestScenario 场景ToViewAndLoadFileBucketSubView]];}每个场景都映射到步骤(要了解更多关于基于测试驱动程序开发的小黄瓜语法和行为驱动开发，我强烈建议阅读这本关于 黄瓜):
/* @given 应用程序处于全新状态@并且用户已经有一个 imap 电子邮件帐户，该帐户具有有效的用户名/密码@then 用户可以成功登录@and 收件箱视图将被加载@并且收件箱将加载用户收件箱中的最新一批电子邮件*/+ (id)scenarioToCompleteSignInAndLoadInbox{KIFTestScenario *scenario =[KIFTestScenario scenarioWithDescription:@"测试一个用户可以成功登录."];[场景 addStepsFromArray:[KIFTestStep stepsCompleteSignInAndLoadInbox]];返回场景；}/* @假设用户已经登录@并且用户已经下载了他们的文件夹@then 用户可以点击文件夹视图@and 用户可以点击附件"远程文件夹@and 将下载附件"远程文件夹中的最新批次*/+ (id)scenarioToFillAttachmentsWithData {KIFTestScenario* 场景 =[KIFTestScenario scenarioWithDescription:@"测试我们可以查看附件文件夹并填写它与数据."];[场景 addStepsFromArray:[KIFTestStep stepsToFillAttachmentsWithData]];返回场景；}/* @假设用户已经登录@并且用户已经下载了他们的文件夹@and 用户已经下载了附件@then 用户可以点击收件箱菜单按钮@and 用户可以点击文件夹列表菜单按钮@并且用户可以单击文件桶图标(在帐户列表视图上)@and 文件桶的数据是从数据库中获取的@and 文件桶视图显示附件*/+ (id)scenarioToViewAndLoadFileBucket {KIFTestScenario *scenario =[KIFTestScenario scenarioWithDescription:@"测试一个用户可以成功查看和加载文件桶父视图"];[场景 addStepsFromArray:[KIFTestStep stepsToViewAndLoadFileBucketPage]];返回场景；}/* @假设用户已经登录@并且用户已经下载了他们的文件夹@and 用户已经下载了附件@并且用户已经打开了文件桶视图@then 用户可以点击文件桶视图表中的随机行@and 子视图将从数据库中检索与该行相关的数据@and subview 将在 uitableview 中显示数据*/+ (id)scenarioToViewAndLoadFileBucketSubView {KIFTestScenario *scenario =[KIFTestScenario scenarioWithDescription:@"测试一个用户可以成功查看和加载文件桶子视图"];[场景 addStepsFromArray:[KIFTestStep stepsToViewAndLoadFileBucketSubPage]];返回场景；}使用 KIF 的 UI 自动化方法定义步骤(这只是一个示例):
//此步骤假设有一个附件文件夹，其中包含带有附件的电子邮件+ (NSArray *)stepsToFillAttachmentsWithData {NSMutableArray* 步骤 = [@[] mutableCopy];【步骤添加对象:[KIFTestStep stepToTapViewWithAccessibilityLabel:@"InboxMenuButton"]];NSIndexPath* indexPath =[NSIndexPath indexPathForRow:remoteAttachmentFolderNumber inSection:0];KIFTestStep* tapAttachmentRowStep =[KIFTestStep stepToTapRowInTableViewWithAccessibilityLabel:@"附件" atIndexPath:indexPath];[步骤添加对象:[KIFTestStep stepToWaitForNotificationName:(NSString *)kBeganSyncingOlderEmails 对象:无whileExecutingStep:tapAttachmentRowStep]];[步骤 addObject:tapAttachmentRowStep];【步骤添加对象:[KIFTestStep stepToWaitForViewWithAccessibilityLabel:@"attachments"]];KIFTestStep *fillingInboxStep =[KIFTestStep stepToWaitForNotificationName:(NSString *)kOldMailBatchDelivered object:nil];[fillingInboxStep setTimeout:kSpecialTimeoutForLongTests];[步骤addObject:fillingInboxStep];返回步骤；}<小时>
KIF 2.0 示例代码:KIF 2.0 使用 Xcode 5 的全新 test navigator.. 这是一个巨大 比 KIF 1.0 所做的改进.. 现在您的测试感觉比过去更加有机和自然..(即它是实时进行的..而不是创建在未来运行的场景等)..你甚至可以用播放按钮等来测试每个..你应该试试看.
这里有一些例子(同样使用 gherkin 语法):
#import <KIF/KIF.h>#import "KIFUITestActor+EXAdditions.h"#import "KIFUITestActor+UserRegistration.h"@interface 登录测试:KIFTestCase@结尾@implementation 登录测试- (void)testReset {[测试仪flushDbase]；[测试仪重置]；}/* @假设应用程序处于全新的干净状态@and 没有人在服务器上注册过@then 用户可以在服务器上注册他们自己@and 立即从骑手的地图开始@and 他们在地图上的位置显示*/- (void)testRegistration{[测试仪flushDbase]；[测试仪重置]；[测试者单用户注册];[测试人员 showUserCurrentLocationOnMap];}/* @假设用户已经注册到服务器@and 用户当前没有登录@then 用户可以使用他们的用户名和密码登录@and 立即从骑手的地图开始@and 他们在地图上的位置显示*/- (void)testSuccessfulLogin{[测试仪重置]；[测试人员登录]；[测试人员 showUserCurrentLocationOnMap];}/* @假设用户已经注册@并且用户在应用程序启动之前已经登录@then 用户从地图视图开始，位置可见@and 按钮提示他们设置接送地点*/- (void)testStartOfApplication {[测试人员 showUserCurrentLocationOnMap];[测试仪显示PickUpButton]；}@结尾下面是分类文件中一些测试用例的实现:
- (void)reset{[自我运行块:^KIFTestStepResult(NSError **错误){BOOL 成功重置 = YES;//为您的应用程序执行实际重置.如果失败，则设置successfulReset = NO.AppDelegate* appDelegate = [[UIApplication sharedApplication] 委托];[appDelegate resetApp];KIFTestCondition(successfulReset, error, @"未能重置应用程序的某些部分.");返回 KIFTestStepResultSuccess；}];}- (void)flushDbase {[自我运行块:^KIFTestStepResult(NSError **错误){NSURL *url = [NSURL URLWithString:@"http://randomdomain.com/flush_db"];NSURLRequest *request = [NSURLRequest requestWithURL:url];NSError *connectionError = nil;BOOL databaseFlushSucceeded = YES;NSURLResponse *响应；NSData *resultData = [NSURLConnection sendSynchronousRequest:request returnedResponse:&response error:&connectionError];如果(！结果数据){databaseFlushSucceeded = 否；KIFTestCondition(databaseFlushSucceeded, error, @"连接服务器失败！");}如果(连接错误){databaseFlushSucceeded = 否；KIFTestCondition(databaseFlushSucceeded, error, [NSString stringWithFormat:@"连接失败.错误: %@", [connectionError localDescription]]);}返回 KIFTestStepResultSuccess；}];}- (void)navigateToLoginPage{[self tapViewWithAccessibilityLabel:@"登录邮箱"];}- (void)returnToLoggedOutHomeScreen{[self tapViewWithAccessibilityLabel:@"Logout"];[self tapViewWithAccessibilityLabel:@"Logout"];//关闭警报.}Note: I added kif to the title just for search indexing puposes, considering that most of the answer turned out to discuss it
I'm looking for something like selenium for iOS, basically a test-automation/unit test framework that can run a certain UI scenario many many times until it crashes, which would help me narrow down the cause of a UI bug that happens very rarely and randomly.
(and by the way, I've NSLogged every single line of code of datasource/table interaction and spent hours analyzing the potential cause.. but found nothing conclusive.. again this bug very rarely happens).
I looked at some of the unit testing frameworks in iOS, but they seem to be so many. I'm not sure which to pick. Also my reference to selenium is based on conjecture, as I've worked with QA folks who've used Selenium in large web projects in the past (and i'm assuming that there must be something similar for iOS).
Now that I'm a one man team working on an iOS project, I'm gonna have to put a QA hat on and figure this bug out.
I'm facing a classic bug that happens when there is a discrepancy between the actual number of rows inserted in a UITableView and the number of rows that the datasource delegate returns. This is the error message:
*** Assertion failure in -[UITableView
_endCellAnimationsWithContext:] Exception in insertRows: Invalid
update: invalid number of rows in section 0.
The number of rows contained in an existing section after the update (2) must be equal to
the number of rows contained in that section before the update (2),
plus or minus the number of rows inserted or deleted from that section
(1 inserted, 0 deleted) and plus or minus the number of rows moved
into or out of that section (0 moved in, 0 moved out).
I click on a UITableViewCell that takes me into another UITableView. Sometimes it works
and sometimes (very rarely) it doesn't (with the above error):
update:.. i've added example code about KIF 2.0 at the bottom after divider.. for those who are more interested in KIF than the specific problem i'm facing:
After some research and experimenting.. I've narrowed down my options to two test-automation libraries:
Frank and KIF. I ultimately decided to use KIF while borrowing cucumber's Gherkin syntax to describe my unit tests.
The reason I chose KIF (rather than Frank) was that KIF is 100% obj-c based, rather than using ruby as well as was the case with Frank. So setting up is simpler, and it was more applicable to my narrow test case requirement. That being said, I admit Frank would be more useful if my application was more complicated (ie using intput from multiple servers etc). You can see the last quarter of this excellent presentation to learn more about the pros and cons of KIF, Frank and other automation-testing frameworks including Apple's own UI Automation.
After using KIF, I found the bug causing the error above, and I could reproduce it using KIF 100% of the time! The reason why it happened so rarely was because it happened only when I tapped through the screens really fast.. and since KIF automates the steps.. it does them at an incredibly fast speed.. which exposed the bug :).
So following will be a sample of the code I used for testing.. this is just to give you a quick feel of what KIF (and Gherkin) can do for you:
in one file I specify the scenarios I want to run:
- (void)initializeScenarios;
{
[self addScenario:[KIFTestScenario scenarioToCompleteSignInAndLoadInbox]];
[self addScenario:[KIFTestScenario scenarioToFillAttachmentsWithData]];
[self addScenario:[KIFTestScenario scenarioToViewAndLoadFileBucket]];
[self addScenario:[KIFTestScenario scenarioToViewAndLoadFileBucketSubView]];
}
each scenario maps to steps (to understand more about the gherkin syntax -and behavioral driven development, which is based on test driver development, I strongly recommend to read this excellent book about cucumber):
/* @given the application is at a fresh state
@and the user already has an imap email account with a valid username/pwd
@then the user can successfully log in
@and the inbox view will be loaded
@and the inbox will get loaded with the latest batch of emails in the user inbox
*/
+ (id)scenarioToCompleteSignInAndLoadInbox
{
KIFTestScenario *scenario =
[KIFTestScenario scenarioWithDescription:@"Test that a user
can successfully log in."];
[scenario addStepsFromArray:[KIFTestStep stepsCompleteSignInAndLoadInbox]];
return scenario;
}
/* @given that the user is already signed in
@and the user has already downloaded their folders
@then the user can click on the folders view
@and the user can click on the 'attachments' remote folder
@and the latest batch from the 'attachments' remote folder will download
*/
+ (id)scenarioToFillAttachmentsWithData {
KIFTestScenario* scenario =
[KIFTestScenario scenarioWithDescription:@"Test that we can view the
attachments folder and fill
it with data."];
[scenario addStepsFromArray:[KIFTestStep stepsToFillAttachmentsWithData]];
return scenario;
}
/* @given that the user is already signed in
@and the user has already downloaded their folders
@and the user has already downloaded attachments
@then the user can click on inbox menu button
@and the user can click on folder list menu button
@and the user can click on the file bucket icon (on the account list view)
@and the data for the file bucket is fetched from the dbase
@and the file bucket view displayes the attachments
*/
+ (id)scenarioToViewAndLoadFileBucket {
KIFTestScenario *scenario =
[KIFTestScenario scenarioWithDescription:@"Test that a user can successfully
view and load
file bucket parent view"];
[scenario addStepsFromArray:[KIFTestStep stepsToViewAndLoadFileBucketPage]];
return scenario;
}
/* @given that the user is already signed in
@and the user has already downloaded their folders
@and the user has already downloaded attachments
@and the user has already opened file bucket view
@then the user can click on a random row in the file bucket view table
@and the subview will retrieve data from the dbase pertaining to that row
@and the subview will display the data in the uitableview
*/
+ (id)scenarioToViewAndLoadFileBucketSubView {
KIFTestScenario *scenario =
[KIFTestScenario scenarioWithDescription:@"Test that a user can successfully
view and load filet
bucket sub view"];
[scenario addStepsFromArray:[KIFTestStep stepsToViewAndLoadFileBucketSubPage]];
return scenario;
}
and steps are defined using KIF's UI automation methods (this is just one example):
// this step assumes there is an attachment folder that contains emails with attachments
+ (NSArray *)stepsToFillAttachmentsWithData {
NSMutableArray* steps = [@[] mutableCopy];
[steps addObject:
[KIFTestStep stepToTapViewWithAccessibilityLabel:@"InboxMenuButton"]];
NSIndexPath* indexPath =
[NSIndexPath indexPathForRow:remoteAttachmentFolderNumber inSection:0];
KIFTestStep* tapAttachmentRowStep =
[KIFTestStep stepToTapRowInTableViewWithAccessibilityLabel:
@"attachments" atIndexPath:indexPath];
[steps addObject:[KIFTestStep stepToWaitForNotificationName:
(NSString *)kBeganSyncingOlderEmails object:nil
whileExecutingStep:tapAttachmentRowStep]];
[steps addObject:tapAttachmentRowStep];
[steps addObject:
[KIFTestStep stepToWaitForViewWithAccessibilityLabel:@"attachments"]];
KIFTestStep *fillingInboxStep =
[KIFTestStep stepToWaitForNotificationName:
(NSString *)kOldMailBatchDelivered object:nil];
[fillingInboxStep setTimeout:kSpecialTimeoutForLongTests];
[steps addObject:fillingInboxStep];
return steps;
}
KIF 2.0 sample code: KIF 2.0 uses Xcode 5's all new test navigator.. which is a huge improvement than what KIF 1.0 was doing.. now your tests feel a lot more organic and natural than the past.. (ie it goes in real time.. rather than creating scenarios that run in the future etc).. you even get to test each one with a play button etc.. you should try it out.
here are some examples (again using gherkin syntax):
#import <KIF/KIF.h>
#import "KIFUITestActor+EXAdditions.h"
#import "KIFUITestActor+UserRegistration.h"
@interface LoginTests : KIFTestCase
@end
@implementation LoginTests
- (void)testReset {
[tester flushDbase];
[tester reset];
}
/* @given that the app is in a fresh clean state
@and that no one has ever registered with the server
@then the user can register their themselves with the server
@and immediately start with the rider's map
@and their location on the map shows
*/
- (void)testRegistration
{
[tester flushDbase];
[tester reset];
[tester singleUserRegistration];
[tester showUserCurrentLocationOnMap];
}
/* @given that the user has already registered with the server
@and the user is not currently logged in
@then the user can login using their user name and password
@and immediately start with the rider's map
@and their location on the map shows
*/
- (void)testSuccessfulLogin
{
[tester reset];
[tester login];
[tester showUserCurrentLocationOnMap];
}
/* @given that the user has already registered
@and that the user is already logged in before app launch
@then the user starts on the map view with the location visible
@and the button prompts them to set pick up location
*/
- (void)testStartOfApplication {
[tester showUserCurrentLocationOnMap];
[tester showsPickUpButton];
}
@end
here is the implementation of some of the test cases in the category files:
- (void)reset
{
[self runBlock:^KIFTestStepResult(NSError **error) {
BOOL successfulReset = YES;
// Do the actual reset for your app. Set successfulReset = NO if it fails.
AppDelegate* appDelegate = [[UIApplication sharedApplication] delegate];
[appDelegate resetApp];
KIFTestCondition(successfulReset, error, @"Failed to reset some part of the application.");
return KIFTestStepResultSuccess;
}];
}
- (void)flushDbase {
[self runBlock:^KIFTestStepResult(NSError **error){
NSURL *url = [NSURL URLWithString:@"http://randomdomain.com/flush_db"];
NSURLRequest *request = [NSURLRequest requestWithURL:url];
NSError *connectionError = nil;
BOOL databaseFlushSucceeded = YES;
NSURLResponse *response;
NSData *resultData = [NSURLConnection sendSynchronousRequest:request returningResponse:&response error:&connectionError];
if (!resultData) {
databaseFlushSucceeded = NO;
KIFTestCondition(databaseFlushSucceeded, error, @"failed to connect to server!");
}
if (connectionError) {
databaseFlushSucceeded = NO;
KIFTestCondition(databaseFlushSucceeded, error, [NSString stringWithFormat:@"connection failed. Error: %@", [connectionError localizedDescription]]);
}
return KIFTestStepResultSuccess;
}];
}
- (void)navigateToLoginPage
{
[self tapViewWithAccessibilityLabel:@"login email"];
}
- (void)returnToLoggedOutHomeScreen
{
[self tapViewWithAccessibilityLabel:@"Logout"];
[self tapViewWithAccessibilityLabel:@"Logout"]; // Dismiss alert.
}
这篇关于KIF:如何自动运行/压力测试 iOS 应用程序以找出罕见 UI 错误的原因?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
