时序问题的不可靠/片状Capybara/AngularJS集成测试 [英] Unreliable/Flakey Capybara/AngularJS Integration Tests With Timing Issues

问题描述

目前这些测试是易燃品.
有时他们过去了.有时它们会失败.
下面是证明此问题的设置，代码和输出.
解决这个问题的建议将不胜感激，我相信会对其他许多人有所帮助，所以请发表评论！

Currently these tests are flakey.
Sometimes they pass. Sometimes they fail.
Below is the setup, code and output demonstrating this issue.
Suggestions to overcome this issue will be greatly appreciated and I am sure will help many others, so please comment!

1. Rails 3.2
2. RSpec 2.x
3. 水豚
4. 团结主义者
5. PhantomJS
6. AngularJS
7. Google Chrome版本47.0.2526.106(64位)

从Gemfile.lock测试宝石

capybara (2.1.0)
database_cleaner (0.7.1)
debug_inspector (0.0.2)
guard-bundler (0.1.3)
guard-livereload (1.2.0)
guard-rspec (2.1.2)
jasminerice (0.0.10)
pg (0.17.1)
phantomjs (2.1.1.0)
poltergeist (1.4.1)
protractor-rails (0.0.17)
pry (0.9.12)
rack (1.4.7)
rack-test (0.6.3)
rails (3.2.21)
rails-assets-angular (1.3.20)
rspec-rails (2.11.4)
simplecov (0.8.2)
sprockets (2.2.3)
zeus (0.13.3)
zeus-parallel_tests (0.2.1)

我尝试过的事情

1. 确保我使用了Capybara的等待DSL匹配器
2. 确保正确设置了数据库清洁程序
3. 假设每个页面项目可能不在页面上并且仍然可以加载，请对其进行测试
4. 缩小不一致的测试
5. 单独运行不一致的测试

6. 识别代码Capybara DSL，它们是导致不一致的测试结果的触发因素.

1. Ensure that I use Capybara's waiting DSL matchers
2. Ensure that my database cleaner is correctly setup
3. Test every page item assuming that it might not be on the page and could still be loading
4. Narrow down inconsistent tests
5. Run inconsistent test alone

6. Identify code Capybara DSLs that are the trigger to inconsistent test results.

• 即创建新记录并假设页面已重定向并且该记录在页面上click_on

或

• .click不能始终正常运行

我使用的资源

[1] 水豚DSL
[2] 水豚，PhantomJs，Poltergeist和Rspec提示
还有更多...

Resources I used

[1] Capybara The DSL
[2] Capybara, PhantomJs, Poltergeist, and Rspec Tips
And many more...

rspec spec/integration/costings/show_costing_spec.rb --format documentation

require "spec_helper"

RSpec.describe "Show a new costing in the listing," do

  before :each do
    admin_sign_in
    create_costing("test1")
  end

  it "shows the costing after creation" do
    within "#costings_table" do
      expect(page).to have_css("#name", text: "test1")
    end
  end

  it "shows the details of the new costing after creation" do
    expect(page).to have_content("Costings")
    within "#costings_table" do
      expect(page).to have_content("test1")
      all("#show").last.click
    end

    expect(page).to have_content("Costing Details")
    expect(page).to have_css("#name", text: "test1")
  end
end  

spec_helper.rb

spec_helper.rb

# This file is copied to spec/ when you run 'rails generate r spec:install'  
ENV["RAILS_ENV"] ||= 'test'
require File.expand_path("../../config/environment", __FILE__)
# Add library functions here so we can test them.
require File.expand_path(File.dirname(__FILE__) + "/../lib/general")
require 'rspec/rails'
require 'rspec/autorun'

# Integration Testing
require 'capybara/poltergeist'
Capybara.register_driver :poltergeist_debug do |app|
 Capybara::Poltergeist::Driver.new(app, :inspector => true)  
end
Capybara.javascript_driver = :poltergeist_debug
Capybara.default_driver = :poltergeist_debug

# Capybara Integration Test Helpers
def admin_sign_in
  visit "/login"
  #Create staff member in database
  Staff.make!(:admin)
  #Log In
  fill_in "staff_username", with: "adminstaff"
  fill_in "staff_password", with: "password"
  click_button "login"
end

def create_costing(item)
  visit "/api#/costings"
  click_on "new_btn"
  within "#form_costing" do
    find("#name", match: :first).set("#{item}")
    find("#description", match: :first).set("test description")    
    find("#from_date", match: :first).set("15/02/2016")
    find("#cost_hourly_cents", match: :first).set("1.00")
    click_on "create_btn"
  end
end

RSpec.configure do |config|
  config.before(:suite) do
    # Requires supporting ruby files with custom matchers and macros, etc,
    # in spec/support/ and its subdirectories.
    require File.expand_path(File.dirname(__FILE__) + "/support/blueprints")
    Dir[Rails.root.join("spec/support/**/*.rb")].each {|f| require f}
  end

  # Remove this line if you're not using ActiveRecord or ActiveRecord fixtures
  config.fixture_path = "#{::Rails.root}/spec/fixtures"

  # Allow a 'focus' tag so that we can run just a few tests which we are currently working on
  config.treat_symbols_as_metadata_keys_with_true_values = true
  config.filter_run focus: true
  config.run_all_when_everything_filtered = true
  config.filter_run_excluding :slow unless ENV["SLOW_SPECS"]

  # Defer Garbage Collection
  config.before(:all) { DeferredGarbageCollection.start }
  config.after(:all)  { DeferredGarbageCollection.reconsider }

  # If you're not using ActiveRecord, or you'd prefer not to run each of your
  # examples within a transaction, remove the following line or assign false
  # instead of true.
  config.use_transactional_fixtures = false
  # config.infer_spec_type_from_file_location!

  # Configure Database Cleaner
  config.include Capybara::DSL
  config.before(:suite) do
    DatabaseCleaner.clean_with(:truncation)
  end

  config.before(:each) do
    DatabaseCleaner.strategy = :transaction
  end

  config.before(:each, :js => true) do
    DatabaseCleaner.strategy = :truncation
  end

  config.before(:each) do
    DatabaseCleaner.start
  end

  config.after(:each) do
    DatabaseCleaner.clean
  end
end

测试结果

Test Run 1: Failing

运行选项: 包括{:focus => true} 排除{:slow => true}

Run options: include {:focus=>true} exclude {:slow=>true}

所有示例均被过滤掉；忽略{:focus => true}

All examples were filtered out; ignoring {:focus=>true}

在清单中显示新的成本核算， 显示创建后的成本核算 显示创建后新成本核算的详细信息(失败-1)

Show a new costing in the listing, shows the costing after creation shows the details of the new costing after creation (FAILED - 1)

失败:

1)在清单中显示新的成本核算，
显示创建后新成本核算的详细信息
失败/错误:expect(page).to have_content("test1")
预期#has_content?("test1")返回true，得到false
#./spec/integration/costings/show_costing_spec.rb:20:
中的块(3个级别) #./spec/integration/costings/show_costing_spec.rb:19:在

1) Show a new costing in the listing,
shows the details of the new costing after creation
Failure/Error: expect(page).to have_content("test1")
expected #has_content?("test1") to return true, got false
# ./spec/integration/costings/show_costing_spec.rb:20:in block (3 levels) in
# ./spec/integration/costings/show_costing_spec.rb:19:in block (2 levels) in

在5.46秒内完成 2个示例，其中1个失败

Finished in 5.46 seconds 2 examples, 1 failure

Test Run 2: Passing

运行选项: 包括{:focus => true} 排除{:slow => true}

Run options: include {:focus=>true} exclude {:slow=>true}

所有示例均被过滤掉；忽略{:focus => true}

All examples were filtered out; ignoring {:focus=>true}

在清单中显示新的成本核算，
显示创建后的成本计算
显示创建后新成本核算的详细信息

Show a new costing in the listing,
shows the costing after creation
shows the details of the new costing after creation

在3.57秒内完成 2个示例，失败0次

Finished in 3.57 seconds 2 examples, 0 failures

更新1

将测试gem升级到以下版本:
水豚(2.6.2)来自(2.1.0)
database_cleaner(1.5.1)from(0.7.1)
debug_inspector(0.0.2)
守卫捆绑手(0.1.3)
Guard-livereload(1.2.0)
Guard-spec(2.1.2)
茉莉(0.0.10)
pg(0.17.1)
phantomjs(2.1.1.0)
来自(1.4.1)的Poltergeist(1.9.0)
量角器(0.0.17)
从(0.9.12)撬起(0.10.3)
机架(1.4.7)
机架测试(0.6.3)
导轨(3.2.21)
rails-assets-angular(1.4.9)from(1.3.20)
(2.11.4)中的 rspec-rails(3.4.2)
simplecov(0.8.2)
链轮(2.2.3)
宙斯(0.13.3)
zeus-parallel_tests(0.2.1)

Update 1

Upgraded testing gems to the following versions:
capybara (2.6.2) from (2.1.0)
database_cleaner (1.5.1) from (0.7.1)
debug_inspector (0.0.2)
guard-bundler (0.1.3)
guard-livereload (1.2.0)
guard-spec (2.1.2)
jasminerice (0.0.10)
pg (0.17.1)
phantomjs (2.1.1.0)
poltergeist (1.9.0) from (1.4.1)
protractor-rails (0.0.17)
pry (0.10.3) from (0.9.12)
rack (1.4.7)
rack-test (0.6.3)
rails (3.2.21)
rails-assets-angular (1.4.9) from (1.3.20)
rspec-rails (3.4.2) from (2.11.4)
simplecov (0.8.2)
sprockets (2.2.3)
zeus (0.13.3)
zeus-parallel_tests (0.2.1)

Result 1

不幸的是，升级这些宝石似乎并没有什么作用，我的测试仍然很不稳定.

Unfortunately upgrading these gems did not seem to make a difference and my tests were still flakey.

更新2

我实现了汤姆·沃尔波尔的建议.确保我的admin_sign_in等待登录完成.

Update 2

I implemented Tom Walpole's suggestions. Ensured that my admin_sign_in waits for sign_in to complete.

还按照汤姆的建议更新了我的database_cleaner设置.

Also updated my database_cleaner setup as Tom suggested.

Result 2

对于我的堆栈，这些更改似乎没有效果.

For my stack these changes did not seem to have an effect.

注意:如果不使用AngularJS，我认为这些更改会有所作为.因此，谢谢汤姆的建议.

Note: If one is not using AngularJS I feel these changes would have made a difference. So thank you Tom for your suggestions.

更新3

我需要获取有关测试运行期间发生的情况的更多信息.在网上有记录日志，使用节省屏幕快照的宝石之类的建议，但我不认为这些方法最省时.我想指定我想在哪里暂停测试并查看变量和表单字段的内容.在浏览器中将是理想的.

Update 3

I needed to get more information on what was happening during my test runs. There are suggestions on the net to log, use screen shot saving gems and the like but I did not feel like these would be the most time efficient. I wanted to specify where I wanted the test to pause at and view the contents of variables and form fields. In a browser would be ideal.

我使用的是
我正在使用"save_and_open_page"和"print page.html"进行调试.

What I Used
I was using "save_and_open_page" and "print page.html" to debug.

我搬到的地方
当我运行RSpec 3.4.2时，我添加了一个调试帮助器方法:

What I Moved To
As I was running RSpec 3.4.2 I added a debug helper method:

rails_helper.rb

rails_helper.rb

def debugit
  puts current_url
  require 'pry'
  binding.pry
end

Result 3

将在控制台中打印URL，并且测试将暂停.在这个阶段，我将能够导航到URL，登录，导航到测试页面并查看Capybara测试的内容.

A URL would be printed in the console and the test would be paused. At this stage I would be able to navigate to the URL, login, navigate to the test page and view what the Capybara test had done.

这使我能够确定在使用水豚的fill_in DSL进行测试时出现问题的根源. 在某些测试运行中，将正确填充字段并提交表单. 在其他情况下，可以正确填写表单，但是单击提交按钮的速度会太快.此处的结果是创建了一条记录，但名称和描述的输入字段未保留.

This enabled me to identify that the source of my problems arose when the test was using capybara's fill_in DSL. In some test runs the fields would be populated correctly and the form would be submitted. In the other scenario the form would be filled in correctly but the submit button would be hit too quickly. The result here is that a record was created but input fields of name and description were not persisted.

更新4

我发现，因为我使用的是AngularJS输入表单和表格，所以AngularJS需要一点时间来绑定到输入字段.如果这次不允许，则不会保存输入数据.

Update 4

I discovered that because I was using AngularJS input forms and tables, AngularJS required a tiny bit of time to bind to the input fields. If it was not allowed this time the input data would not be saved.

水豚提供等待方法，例如内部"和查找".我使用了这些，但它们并没有解决AngularJS绑定时间问题. 我发现ng-if可用于创建if语句，以等待特定的项目，这表示AngularJS绑定到表单字段完成.

Capybara provides waiting methods such as "within" and "find". I used these but they did not help with the AngularJS binding time issue. I found ng-if could be used to create an if statement to wait for particular item that would signify the AngularJS bindings to the form fields complete.

因此，我使用Capybara等待方法来等待要填充的字段，并使用AngularJS的ng-if直到它们准备好才显示这些字段.

So I used Capybara waiting methods to wait for the fields I wanted to fill and I used AngularJS' ng-if not to show the fields until they are ready.

实施
index.html.erb

Implementation
index.html.erb

<div  ng-if="tableParams.data">
  <table id="costings_table ng-table="tableParams" class="table">
    <td id="field1">{{table.field1}}</td>
    <td id="field2">{{table.field2}}</td>
  </table>
</div>

Result 4

测试终于通过！但是，我拥有所有带有xpath的find方法，以确保等待特定且难以定位的项目...

Tests finally pass! However I have all these find methods with xpath ensuring specific and difficult to target items are waited on...

更新5

即使在我的gemfile中，我正在运行gem phantomJS版本2.1.1，我的命令行版本也仅为1.X.事实证明这很重要.

Update 5

Even though in my gemfile I was running the gem phantomJS version 2.1.1 my command line version was only 1.X. This proved to be significant.

我将我的命令行phantomJS版本更新为2.1.1.同时，我确保所有输入框，按钮，表格，标题都具有唯一的ID.然后，我可以删除所有find(:xpath)事件，而无需破坏测试.

I updated my command line phantomJS version to 2.1.1. At the same time I ensured all my input boxes, buttons, tables, headings had unique ids. I then was able to remove all find(:xpath) occurances without breaking the tests.

Result 5

这套测试现在可以一直可靠地通过！正是我想要的！是的！

This suite of tests now reliably pass all the time! Exactly what I wanted! Yes!

推荐答案

问题

当测试使用水豚的fill_in DSL时出现问题.在某些测试运行中，将正确填充字段并提交表单.在其他情况下，可以正确填写表单，但是单击提交按钮的速度会太快.此处的结果是创建了一条记录，但名称和描述的输入字段未保留.

The Issue

Problems arose when the test was using capybara's fill_in DSL. In some test runs the fields would be populated correctly and the form would be submitted. In the other scenario the form would be filled in correctly but the submit button would be hit too quickly. The result here is that a record was created but input fields of name and description were not persisted.

需要使用AngularJS的ng-if语句，直到准备就绪，才显示表单字段.
这需要结合使用Capybara等待方法来完成，以确保仅在表单加载完成后才提交fill_in字段.

AngularJS' ng-if statements needed to be used not to show form fields until they are ready.
This needed to be done in conjunction with the use of Capybara waiting methods to ensure fill_in field are only submitted once form load has completed.

index.html.erb或等效文件:

<div  ng-if="tableParams.data">
  <table id="costings_table ng-table="tableParams" class="table">
    <td id="field1">{{table.field1}}</td>
    <td id="field2">{{table.field2}}</td>
   </table>
</div>

2.将PhantomJS的命令行版本更新为最新版本(2.1.1)

这似乎使测试无需太多的Capybara等待方法即可运行，以实现可靠的测试.

2. Updated the command line version of PhantomJS to the latest (2.1.1)

This seemed to enable tests to run without as many Capybara waiting methods in order to achieve reliable tests.

更新的测试代码
show_costing_spec.rb

Updated Test Code
show_costing_spec.rb

require "rails_helper"

RSpec.describe "Show a new costing in the listing,", :type => :feature do

  before :each do
    admin_sign_in
    create_costing("test1")
  end

  it "shows the costing after creation" do
    within "#costings_table" do
      expect(page.find("#code2")).to have_content("2")
      expect(page.find("#name2")).to have_content("test1")
    end
  end

  it "shows the details of the new costing after creation" do
    within "#costings_table" do
      click_on "show2"
    end

    expect(page.find("#page_title")).to have_content("Costing Details")
    expect(page.find("#code")).to have_content("2")
    expect(page.find("#name")).to have_content("test1") 
    expect(page.find("#description")).to have_content("test description")
  end
end

rails_helper.rb

rails_helper.rb

# This file is copied to spec/ when you run 'rails generate rspec:install'
ENV["RAILS_ENV"] ||= 'test'
require File.expand_path("../../config/environment", __FILE__)

# Add library functions here so we can test them.
require File.expand_path(File.dirname(__FILE__) + "/../lib/general")

require 'rspec/rails'
require 'devise'

RSpec.configure do |config|
  config.before(:suite) do
    # Requires supporting ruby files with custom matchers and macros, etc,
    # in spec/support/ and its subdirectories.
    require File.expand_path(File.dirname(__FILE__) + "/support/blueprints")
    Dir[Rails.root.join("spec/support/**/*.rb")].each {|f| require f}

    # Setup Devise before it is used in rails_helper
    config.include Devise::TestHelpers, :type => :controller
    Devise.stretches = 1 # Improves speed.
end

config.include Capybara::DSL, :type => :feature
  config.mock_with :rspec

# Remove this line if you're not using ActiveRecord or ActiveRecord fixtures
config.fixture_path = "#{::Rails.root}/spec/fixtures"

# Allow a 'focus' tag so that we can run just a few tests which we are currently working on
config.filter_run focus: true
config.run_all_when_everything_filtered = true
config.filter_run_excluding :slow unless ENV["SLOW_SPECS"]

# Defer Garbage Collection
config.before(:all) { DeferredGarbageCollection.start }
config.after(:all)  { DeferredGarbageCollection.reconsider }

# Integration Testing
require 'capybara/rspec'
require 'capybara/poltergeist'

Capybara.register_driver :poltergeist_debug do |app|
  Capybara::Poltergeist::Driver.new(app, {:inspector => true, js_errors: false })  
end

Capybara.javascript_driver = :poltergeist_debug
Capybara.default_driver = :poltergeist_debug

# Debugging tools
def debugit
  puts current_url
  require 'pry'
  binding.pry
end

# If you're not using ActiveRecord, or you'd prefer not to run each of your
# examples within a transaction, remove the following line or assign false
# instead of true.
config.use_transactional_fixtures = false

#Show Deprications As Errors with full backtracing
config.raise_errors_for_deprecations!

#rest of the file....
# Final part of Configure Database Cleaner

Capybara.default_max_wait_time = 5
config.use_transactional_fixtures = false

config.before(:suite) do
  if config.use_transactional_fixtures?
    raise(<<-MSG)
      Delete line `config.use_transactional_fixtures = true` from
      rails_helper.rb (or set it to false) to prevent uncommitted
      transactions being used in JavaScript-dependent specs. During 
      testing, the app-under-test that the browser driver connects to 
      uses a different database connection to the database connection 
      used by the spec. The app's database connection would not be 
      able to access uncommitted transaction data setup over the 
      spec's database connection.
     MSG
  end
  DatabaseCleaner.clean_with(:truncation)
end  

config.before(:each) do
  DatabaseCleaner.strategy = :transaction
end

config.before(:each, type: :feature) do
  # :rack_test driver's Rack app under test shares database connection
  # with the specs, so continue to use transaction strategy for speed.
  driver_shares_db_connection_with_specs = Capybara.current_driver == :rack_test

    if !driver_shares_db_connection_with_specs
      # Driver is probably for an external browser with an app
      # under test that does *not* share a database connection with the
      # specs, so use truncation strategy.
      DatabaseCleaner.strategy = :truncation
    end
  end

  config.before(:each) do
    DatabaseCleaner.start
  end

  config.append_after(:each) do
    DatabaseCleaner.clean
  end
end


def admin_sign_in
  visit "/login"

  #Create staff member in database
  Staff.make!(:admin)

  #Log In
  fill_in "staff_username", with: "adminstaff"
  fill_in "staff_password", with: "password"
  click_button "login"

  expect(page).to have_text('Logout')
end

def create_costing(item)
  @item = item
  visit "/api#/costings"

  expect(page).to have_selector("#new_btn")
  click_on "new_btn"

  expect(page).to have_text("New Costing")
  within "#form_costing" do
    fill_in "name", with: "#{@item}"
    fill_in "description", with: "test description"
    fill_in "from_date1", with: "15/02/2015" 
    fill_in "cost_hourly_cents1", with: "12.00"

    expect(page).to have_selector("#create_btn")
    click_on "create_btn"
  end
  expect(page.find("#page_title")).to have_content("Costings")
end

这篇关于时序问题的不可靠/片状Capybara/AngularJS集成测试的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！