Why use Regression Pack Automation?

A regression pack is a collection of test cases performed as each new release/software update is completed.

Regression packs are the most common set of test cases that are always executed post deployment.

They are one of the ‘quality gates’ Quality Assurance (QA) testers use to reassure confidence of a stable deployment. Regression testing can be done manually, or it can be automated.

Switching from manual to automated regression testing

Is it essential to have Regression Pack Automation? Although it's feasible to have manual QA’s completing regression testing on the Sitecore websites we build, it does take longer and increases the likelihood of human error. Both of these can be reduced drastically by using automation, as machines are less prone to errors and can run test cases significantly faster than humans.

Automated regression testing significantly improves efficiency during crucial time restricted deployments.

Demand for automation has increased due to organisations moving towards a Continuous Integration approach with their deployments. This allows regression testing to continue while QAs perform exploratory or cross-browser testing in parallel to new Sitecore features that have been deployed.

Automation test case

To demonstrate the efficiency of automation, the Codehouse QA team created an automated regression pack for our Sitecore website.

Manual testing took an average of an hour QA. but with automated regression testing it only took 15 minutes; 10 minutes for the test scripts to run and 5 minutes to validate if the failing tests were bugs (valid failures).

If this test case is simulated across a deployment with multiple servers, for example 2 servers which each have 2 Content Delivery (CD) boxes then the regression test would need to be repeated 4 times per CD.

Automated regression test case

A time saving of 3 hours is achieved. This drastically reduces the QA testing time during a deployment.

Automation: Things to consider

The main considerations with automation is the setup period & potential UI changes affecting the pass rate % per deployment:

Setup

The setup period is the duration required to automate the candidate test cases from the manual regression pack.

This can vary on the number of test cases and the intricacy of the web pages visitors are navigating between i.e. heavy JS loaded pages or iFrames within the HTML.

Maintenance & validating failures

It's likely UI changes or new functionality will break test cases as they need to be refactored with updated HTML attributes. If this is left untouched in post deployments, more time will be spent refactoring during deployments.

Automation tools

Of the many web automation tools available, Selenium, is our preferred choice.

Selenium isn’t restricted to one specific platform or application. It was developed to work in multiple variations depending on the desired outcome for automation.

Benefits of Selenium automation tool

  • As it’s open source it has a strong and supportive online community to help with the majority of Selenium related issues
  • Supports cross-language i.e. it’s not limited to one language as it can be written in Java, C#, Ruby, JavaScript etc.
  • It has cross-OS Support  (Windows, Mac and Linux)
  • If offers cross browser support
  • A variety of reporting tools can be implementation to test cases
  • It can be used to automate websites, mobile sites, applications and APIs.
  • Time duration can be reduced further by using Selenium Grid to run multiple test cases in parallel rather than sequential
Silenium

Running real world test scenarios

With automation on the rise, writing test scripts is becoming a skillset all QA’s should have. At Codehouse, we’ve written several custom test scenarios for customers running Sitecore websites.

Regression testing is automated through Selenium and run every time there’s a Sitecore deployment through Octopus Deploy. Post execution, the automated script emails a high-level report consisting of:

  • Total number of pass and fail features
  • Total number of pass and fail test scripts
  • The overall percentage pass rate
  • List of features executed
  • Feature overview - if the test passed it is highlighted as “Passed”. If however a test “failed” within a feature, it will share the error log indicating the point of failure upon clicking the selected feature
Regression test results

Benefits of Regression Pack Automation

  • Less time required for completing the regression testing per environmental deployment
  • Potential to configure the automated regression packs starting from the customer sandbox environments to production
  • QA resource not mandatory to give QA sign off, as the automated test run generates a test report. However, if there are issues in the test report, QA’s will need to investigate before giving sign off
  • Flexibility to run regression testing out of hours if necessary, as scripts can be executed on Octopus without direct QA involvement other than verifying the scripts the next working day

Regression Pack Automation, the cloud & ROI

Even though manual testing is still a valid alternative, automated testing is on the increase, with many tech businesses moving to the cloud. Preparation is key in ensuring test scenarios are ready for automation during regression testing.

ROI too is important when considering Regression Pack Automation, as automated testing during the early stages of development yields positive ROI.

Codehouse and Regression Pack Automation

Our expert Quality Assurance team is dedicated to streamlining the deployment process. We've helped many customers spanning diverse industries with this approach. Contact us to find out more.