Which UI tests should we automate?

Our website front-end development team is moving to a new open source JavaScript platform and wanted our end-to-end (E2E) automated tests written in the same language, which means a rewrite from C# into JavaScript.  This gives our test team a chance at a do-over when it comes to test automation.

We have a lot of existing automated tests, but…

For sure we have a large number of automated tests that cover all layers of the stack including unit tests, API tests, and UI tests.  These tests number in the thousands.  But taking a step back to analyze our entire suite, a number of questions present themselves:

  • What’s our site coverage?
  • How many tests should we have?
  • If 20% of our tests are flaky or routinely fail, what should we do with them?
  • Are we staffed to analyze all test failures?
  • Are we missing any key customer scenarios?
  • Are we executing tests for functionality that customers rarely use?

Role of automated testing

The test pyramid, described by Mike Cohn in his book Succeeding with Agile: Software Development Using Scrum, consists of three layers.  It’s hard to disagree with the opinion that UI tests should comprise the smallest component of automated testing.  They are the slowest to run, and the most expensive to write and maintain.  That being said, they are great at verifying the user’s experience.

Test Pyramid

When should UI tests be automated?

As a UI test automator, it’s not uncommon for me to encounter those who flatly rule out any benefits for UI automation.  But on our team, we always validate new code deployments and any tests not automated have to be manually executed.  It’s interesting that those opposing the idea of UI automation aren’t exactly volunteering to run the manual regressions either.

I understand the reluctance to develop UI automation; careful consideration of the following factors is important to prevent short-lived throwaway work.

  • Functionality should always be understood and validated manually  first
  • A test should not be too difficult (expensive) to automate
  • The team should be staffed with skilled automation developers
  • High priority customer scenarios are identified
  • The product code is stable and most tests won’t need constant updating
  • The test team is staffed to analyze test results

Tests that shouldn’t be automated

Some tests of course will not be good candidates for automation:

  • Unlikely corner cases
  • Exploratory tests (how would you automate those?)
  • Testing for visual appearance or layout

Planning automated regression tests from a customer perspective

Our existing UI automated test suite provides decent coverage of the primary happy-path workflows on our site.  They cover the main operations typical on most storefront sites:  creating accounts, resetting passwords, making purchases, and viewing product information.  At the time we created them we thought our coverage was pretty good.

What we failed to take into account was real data on how customers used our site. Basically we guessed.

Once we had an incident over a holiday where one of our purchase paths was completely broken for three days without setting off any alarms.  After the problem was fixed, we started putting together a strike team to close the test coverage gap.  To our surprise we were made aware that the product was rarely purchased on our site, with the exception of purchases made by the test team during validation!

Lesson learned.  In the future we’ll make sure and get real customer usage data before designing our test coverage.  We’ve identified several sources to get this data.

Who better than the customer telephone support team would be aware of critical paths actually used by our site visitors.  We’ve got to those relationships started and maintained.

Another source we should have been consulting is Google Analytics data.  The business analysts already have the data on which pages get the most traffic yet our QA team had not taken the time to consider it.   Seems like a major oversight.

Ideal future state of our regression test suite

Our test leads put their heads together and came up with some key ideas to remember going forward:

  • Automated tests should be stored with the site code where developers can easily find them and update them
  • The test case priorities should be driven by:
    • Customer site usage data
    • Customer support team input — what are the biggest customer pain points?
    • High visibility bug regression tests.
    • Workflows and site paths ranked by revenue histories
  • The test suite should not be driven by:
    • Tests that are easy to automate
    • Guesses as to the most used parts of the site
    • Development team acceptance test automation only (it’s incomplete)
  • Tests must be able to run in any environment including production
  • Test transactions must be identifiable for easy cancellation/reversal

Conclusion

UI automated tests do a great job of verifying the site is working from a customer’s perspective.  Before you tackle the planning for your test suite, take a step back and gather the usage data you need to get the highest impact from your tests.  You’ll sleep better and have a great story for management.

 

A Clean and DRY C#/Selenium Test Framework

In this post I share a UI test framework design that’s worked well for our automation team.  This framework design strives to adhere to the don’t repeat yourself (DRY) principle from the book The Pragmatic Programmer and I think this is reflected in the way we structured the solution.

I created a little C# test project to demonstrate the simplicity of the design.  It contains a single test method that opens a Chrome browser, navigates to the Yahoo search page, enters a search term, and asserts an expected search result.

Create the Projects

This solution was created in Visual Studio 2013 and contains two projects:

  • Tests (a Unit Tests project)
  • PageMaps (a Class Library project)

Also needed are libraries installed via the Nuget Package Manager which should be added to both projects after you create them:

  • Selenium.WebDriver
  • WebDriver.ChromeDriver.win32
Basic Selenium Test Project
Organization of Visual Studio Solution

To start the first project, open Visual Studio and create a new Unit Test Project named “Tests”:

Unit Test Project
Unit Test Project

Add a second project to the solution, a class library project called “PageMaps”:

Class Library Project
Class Library Project

Create the First Page Map

The big idea is to keep the details of each web page interaction in its own class and separate from the test logic.  This allows some real advantages.

The first advantage is that the page element navigation can be defined once then reused by any test.  If the UI changes, which it always does, the code update is needed only in one location, the page map file. The second advantage is the test method itself can be kept clean and short.

Create a resources file to store the element Xpaths

The Selenium WebDriver component can locate the page elements using several methods, by element Id, XPath, Css, and several others.  I use XPath because it is reliable, doesn’t rely on visible text, and you get the rich Xpath syntax if you want to customize the XPath returned by the Chrome developer tools; yes, Chrome does the work for you.

I like to store my Xpath strings in one place for easy reviewing and updates.  A resources (.resx) file is a great place for this.  Highlight the PageMaps project in the Solution explorer, right-click the name and select Properties.  Select the Resources tab and click on the link shown to create a default resources file.

Add a Resources file

To get the first element XPath for our sample which automates the Yahoo search page, open Chrome to http://search.yahoo.com, right-click the Search button, and click “Inspect Element”.

Inspect element

This opens the Chrome Developer Tools sidebar with the <input> element highlighted.  Right-click the highlighted element and click “Copy XPath”.  Create a string value in the resources file for each needed page element XPath expression as shown below.

Page Maps Properties

Now we’re ready to write some code!

Search Page Map

Add a new class file to the PageMaps project called SearchPage.  We’ll add three properties to wrap the Selenium selectors, one each for the search page text input, the search button, and the search results area.  Type the following code into your SearchPage class.  You’ll need to add a “using OpenQA.Selenium” statement to the top of the file.

 

Page Map Code

We need a method in our SearchPage file that takes a search term string and validates whether the page displays expected text.  If the test succeeds, a reference to our map page is returned, else an Assert exception is thrown.

Submit Search Method

Add a Test Method

Now we’re ready to create our first test.  In the UnitTest1.cs file, which I renamed to SearchTests.cs, change the TestMethod1 name to “BasicSearchShowsWikipediaEntry” and add the statement shown.

Test Method

This test may look a little strange if you’re used to the Arrange–Act–Assert style.  After some back and forth discussion our team settled on this style, aka “fluent”, wherein each method returns the appropriate type of object for the next method.  For more discussion of the fluent style, check out this post by Tom Fischer, Fluent Code in C#.

In this framework, the SubmitSearch method enters the search term, clicks the submit button, asserts the text is found on the page, and returns a reference to the search page.  While the reference isn’t used in this example, a little imagination can enable you to visualize how a multi-step scenario could be done in a single statement by chaining together methods.

For example, imagine for some reason we wanted to test whether 5 consecutive searches all returned results containing a Wikipedia entry.  We could write our test like this:

Another test method

This is a contrived example but imagine you needed to visit several pages in sequence, and each action method returned the landing page to perform the next step.  All the action details and assertions are contained on the map page and the test becomes merely a sequence of steps that can be chained together or tested separately in their own test method.

Conclusion

We’ve been using this “fluent” style of writing test automation for almost a year and it’s been very efficient.  Using a resource file for element paths and having a separate page map for each page under test has also proved its worth. You could extend this example in several ways, such as creating a base map page containing common elements, such as navigation controls, which is inherited by all page maps.

I hope you found these concepts useful.