Which UI tests should we automate?

Our website front-end development team is moving to a new open source JavaScript platform and wanted our end-to-end (E2E) automated tests written in the same language, which means a rewrite from C# into JavaScript.  This gives our test team a chance at a do-over when it comes to test automation.

We have a lot of existing automated tests, but…

For sure we have a large number of automated tests that cover all layers of the stack including unit tests, API tests, and UI tests.  These tests number in the thousands.  But taking a step back to analyze our entire suite, a number of questions present themselves:

  • What’s our site coverage?
  • How many tests should we have?
  • If 20% of our tests are flaky or routinely fail, what should we do with them?
  • Are we staffed to analyze all test failures?
  • Are we missing any key customer scenarios?
  • Are we executing tests for functionality that customers rarely use?

Role of automated testing

The test pyramid, described by Mike Cohn in his book Succeeding with Agile: Software Development Using Scrum, consists of three layers.  It’s hard to disagree with the opinion that UI tests should comprise the smallest component of automated testing.  They are the slowest to run, and the most expensive to write and maintain.  That being said, they are great at verifying the user’s experience.

Test Pyramid

When should UI tests be automated?

As a UI test automator, it’s not uncommon for me to encounter those who flatly rule out any benefits for UI automation.  But on our team, we always validate new code deployments and any tests not automated have to be manually executed.  It’s interesting that those opposing the idea of UI automation aren’t exactly volunteering to run the manual regressions either.

I understand the reluctance to develop UI automation; careful consideration of the following factors is important to prevent short-lived throwaway work.

  • Functionality should always be understood and validated manually  first
  • A test should not be too difficult (expensive) to automate
  • The team should be staffed with skilled automation developers
  • High priority customer scenarios are identified
  • The product code is stable and most tests won’t need constant updating
  • The test team is staffed to analyze test results

Tests that shouldn’t be automated

Some tests of course will not be good candidates for automation:

  • Unlikely corner cases
  • Exploratory tests (how would you automate those?)
  • Testing for visual appearance or layout

Planning automated regression tests from a customer perspective

Our existing UI automated test suite provides decent coverage of the primary happy-path workflows on our site.  They cover the main operations typical on most storefront sites:  creating accounts, resetting passwords, making purchases, and viewing product information.  At the time we created them we thought our coverage was pretty good.

What we failed to take into account was real data on how customers used our site. Basically we guessed.

Once we had an incident over a holiday where one of our purchase paths was completely broken for three days without setting off any alarms.  After the problem was fixed, we started putting together a strike team to close the test coverage gap.  To our surprise we were made aware that the product was rarely purchased on our site, with the exception of purchases made by the test team during validation!

Lesson learned.  In the future we’ll make sure and get real customer usage data before designing our test coverage.  We’ve identified several sources to get this data.

Who better than the customer telephone support team would be aware of critical paths actually used by our site visitors.  We’ve got to those relationships started and maintained.

Another source we should have been consulting is Google Analytics data.  The business analysts already have the data on which pages get the most traffic yet our QA team had not taken the time to consider it.   Seems like a major oversight.

Ideal future state of our regression test suite

Our test leads put their heads together and came up with some key ideas to remember going forward:

  • Automated tests should be stored with the site code where developers can easily find them and update them
  • The test case priorities should be driven by:
    • Customer site usage data
    • Customer support team input — what are the biggest customer pain points?
    • High visibility bug regression tests.
    • Workflows and site paths ranked by revenue histories
  • The test suite should not be driven by:
    • Tests that are easy to automate
    • Guesses as to the most used parts of the site
    • Development team acceptance test automation only (it’s incomplete)
  • Tests must be able to run in any environment including production
  • Test transactions must be identifiable for easy cancellation/reversal


UI automated tests do a great job of verifying the site is working from a customer’s perspective.  Before you tackle the planning for your test suite, take a step back and gather the usage data you need to get the highest impact from your tests.  You’ll sleep better and have a great story for management.



Use Mind Maps Daily to Keep Your Technical Skills Sharp

Workplace changes were forcing me to learn yet another new technology and I was feeling a bit discouraged.  Then I had one of those “ah ha” moments when I remembered how easily I prepared for my MCSE exams a decade earlier using mind maps.  For some reason I quit using them; I realized I could apply those same learning skills again.

During those days when the MCSE (Microsoft Certified System Engineer) certification was so much in vogue, I had determined to get certified through self study.  I pulled it off in about a year of effort; at that time it consisted of six exams on subjects such as networking and SQL Server administration.  It cost me nothing but time as my employer paid for the exams.  I  had a set of study books and mind mapped every chapter.

I think most readers have at least some familiarity with what a mind map is.  Here’s a mind map I created recently from a chapter in a book about blogging.

Mind map about blogging

Tony Buzan, the inventor of mind mapping, says:

When you use Mind Maps on a daily basis, you will find that your life becomes more productive, fulfilled, and successful on every level. There are no limits to the number of thoughts, ideas and connections that your brain can make, which means that there are no limits to the different ways you can use Mind Maps to help you.

Six Benefits of Mind Mapping

1.  Mind-mapping helps you focus when studying

We all do it. We read half a page and then realize we haven’t absorbed a word. When your mind is actively searching for a key concept to add a node on a mind map, it engages your brain to pay attention.

2.  Mind-mapping forces you to pick out the big ideas

Most technical books are logically organized into chapters and sections. I usually make the chapter title the central node of the map and then create major sub-nodes from the section headings.  If there is material that doesn’t interest me I skip it.  I then scan each section for sub-sections and key points.  These become third and fourth tier nodes.

It’s similar to taking notes the traditional way except it’s nonlinear.  You can jump around in the chapter, add auxiliary ideas, go back, rearrange nodes, insert images, icons or change colors to emphasize key ideas.

3.  Mind-mapping creates hooks for your memory

Instead of pages of notes, mind maps are made up of keywords and short phrases, which are easier to remember. By linking each word to the next, you create a natural association, or hook, between concepts.  Adding images and color, your notes aren’t a page of drab sameness.  If preparing for an exam, I find myself able to remember the overall shape and color of a mind map, and usually the details also emerge from the recesses of my memory.

In fact, as I write this, I can still visualize some of the maps I created for the MCSE exams some years ago.

4.  Mind-mapping is fun

You can be creative while you learn.  Have fun adding images that spark your memory.  Adding images and color is analogous to bolding and highlighting in traditional note taking.

5.  Mind-mapping activates different learning styles

Learning styles are generally grouped as visual, auditory, and kinesthetic (physical).  When you create a mind map while you learn new material, you certainly activate your visual learning style.  But you also engage your “hands-on” learning as you draw and create your mind map “picture” using paper or a computer.

6.  Mind-mapping benefits are proven

During that MCSE study period I used paper and colored pens to create mind maps. Now I use a program called SimpleMind™.  There may be free tools, but the price is reasonable and SimpleMind™ works on all my devices: Mac, Windows, iPad, and iPhone.  It uses Dropbox for network storage so I can edit or view my latest mind maps from anywhere.  The example map shown was created using this tool, which I have no financial incentive to promote by-the-way.

I have a pattern of learning a useful technique and then forgetting or dropping it in pursuit of the next shiny thing. This time I’m determined to remember past successes and capitalize on the proven worth of mind maps to continue getting the benefits.

A Clean and DRY C#/Selenium Test Framework

In this post I share a UI test framework design that’s worked well for our automation team.  This framework design strives to adhere to the don’t repeat yourself (DRY) principle from the book The Pragmatic Programmer and I think this is reflected in the way we structured the solution.

I created a little C# test project to demonstrate the simplicity of the design.  It contains a single test method that opens a Chrome browser, navigates to the Yahoo search page, enters a search term, and asserts an expected search result.

Create the Projects

This solution was created in Visual Studio 2013 and contains two projects:

  • Tests (a Unit Tests project)
  • PageMaps (a Class Library project)

Also needed are libraries installed via the Nuget Package Manager which should be added to both projects after you create them:

  • Selenium.WebDriver
  • WebDriver.ChromeDriver.win32
Basic Selenium Test Project
Organization of Visual Studio Solution

To start the first project, open Visual Studio and create a new Unit Test Project named “Tests”:

Unit Test Project
Unit Test Project

Add a second project to the solution, a class library project called “PageMaps”:

Class Library Project
Class Library Project

Create the First Page Map

The big idea is to keep the details of each web page interaction in its own class and separate from the test logic.  This allows some real advantages.

The first advantage is that the page element navigation can be defined once then reused by any test.  If the UI changes, which it always does, the code update is needed only in one location, the page map file. The second advantage is the test method itself can be kept clean and short.

Create a resources file to store the element Xpaths

The Selenium WebDriver component can locate the page elements using several methods, by element Id, XPath, Css, and several others.  I use XPath because it is reliable, doesn’t rely on visible text, and you get the rich Xpath syntax if you want to customize the XPath returned by the Chrome developer tools; yes, Chrome does the work for you.

I like to store my Xpath strings in one place for easy reviewing and updates.  A resources (.resx) file is a great place for this.  Highlight the PageMaps project in the Solution explorer, right-click the name and select Properties.  Select the Resources tab and click on the link shown to create a default resources file.

Add a Resources file

To get the first element XPath for our sample which automates the Yahoo search page, open Chrome to http://search.yahoo.com, right-click the Search button, and click “Inspect Element”.

Inspect element

This opens the Chrome Developer Tools sidebar with the <input> element highlighted.  Right-click the highlighted element and click “Copy XPath”.  Create a string value in the resources file for each needed page element XPath expression as shown below.

Page Maps Properties

Now we’re ready to write some code!

Search Page Map

Add a new class file to the PageMaps project called SearchPage.  We’ll add three properties to wrap the Selenium selectors, one each for the search page text input, the search button, and the search results area.  Type the following code into your SearchPage class.  You’ll need to add a “using OpenQA.Selenium” statement to the top of the file.


Page Map Code

We need a method in our SearchPage file that takes a search term string and validates whether the page displays expected text.  If the test succeeds, a reference to our map page is returned, else an Assert exception is thrown.

Submit Search Method

Add a Test Method

Now we’re ready to create our first test.  In the UnitTest1.cs file, which I renamed to SearchTests.cs, change the TestMethod1 name to “BasicSearchShowsWikipediaEntry” and add the statement shown.

Test Method

This test may look a little strange if you’re used to the Arrange–Act–Assert style.  After some back and forth discussion our team settled on this style, aka “fluent”, wherein each method returns the appropriate type of object for the next method.  For more discussion of the fluent style, check out this post by Tom Fischer, Fluent Code in C#.

In this framework, the SubmitSearch method enters the search term, clicks the submit button, asserts the text is found on the page, and returns a reference to the search page.  While the reference isn’t used in this example, a little imagination can enable you to visualize how a multi-step scenario could be done in a single statement by chaining together methods.

For example, imagine for some reason we wanted to test whether 5 consecutive searches all returned results containing a Wikipedia entry.  We could write our test like this:

Another test method

This is a contrived example but imagine you needed to visit several pages in sequence, and each action method returned the landing page to perform the next step.  All the action details and assertions are contained on the map page and the test becomes merely a sequence of steps that can be chained together or tested separately in their own test method.


We’ve been using this “fluent” style of writing test automation for almost a year and it’s been very efficient.  Using a resource file for element paths and having a separate page map for each page under test has also proved its worth. You could extend this example in several ways, such as creating a base map page containing common elements, such as navigation controls, which is inherited by all page maps.

I hope you found these concepts useful.