Test Automation Case Study: Cucumber, Selenium and Jira Xray

If you have any experience with QA and software testing, you may well already be familiar with Cucumber, Selenium and Xray. But just in case you aren’t, let’s kick things off with some very concise definitions before we move onto a case study on how we applied these technologies and approaches in setting up a test automation system for our partner:

Cucumber is a software testing approach that uses the Gherkin language to simply explain the behaviour of a given application in standard English. It supports a Behaviour Driven Development process to building applications.

Selenium Webdriver is a browser automation tool primarily used for test automation of web-based applications. It can also be used to automate web-based administration tasks.

Xray for Jira is a Test Management app native to the project tracking software. It is designed to manage the entirety of the testing cycle from test planning to design, execution and test reporting. It uses a special type of Jira issue so any professional used to using the software will quickly feel at ease with the system.

What should an organisation achieve when setting up a QA testing process that involves test automation? The following case study illustrates a real-life example of how K&C QA consultants set up a test automation process based on Cucumber, Selenium and Jira Xray.

Agile testing, preferably automated where possible, is the core of any quality assurance (QA) methodology. Our partner is a prominent European e-commerce consumer electronics retailer. Their goal was to ensure and improve the quality of their development process and its results.

At the point we began working with the company it didn’t have any dedicated QA engineers or management. Developers tested own work under their own initiative. This was leading to inconsistent quality and costing the company time and money.

Lacking the in-house resource of dedicated QA engineers, they turned to Krusche & Company’s 20+ years of experience.

The requirement was a stable QA testing process that could be managed, maintained and executed in one place and without documents created in the usual text editors.

We were assigned with two priority tasks:

1. Organize test cases using Cucumber and Selenium within the Xray tool (a Jira plugin for test management).
2. Optimisation of the test automation process.

The roadmap to achieving these two end results involved:

• Creating the QA documentation from scratch.
• Deciding upon and setting up a process to maintain test plans and test execution across different environments.
• Integration of Cucumber and Selenium test automation inside the Xray system.
• Setting up the tests that needed to be executed from Xray.

A further requirement was that an introduction to the system and not a QA background should be all that was necessary. Product Owners or Feature Owners had to be able to create a testing plan using existing test cases.

At the beginning of the project we were introduced to the team. They already worked according to agile principles and documentation for features and tasks was clear and thorough.

But each developer was responsible for testing their own code. There was no one directly responsible for testing, no defined process the developers had to follow and no ultimate responsibility.

A key issue we highlighted was the lack of any dedicated User Acceptance Testing phase. Our partner did have an automation engineer that had created a series of test cases for execution. However, he had to real framework to work within and testing was only executed from the local environment.

Reporting consisted of short chat message that testing had been passed or failed and a list of issues that were created in Jira.

As a first step and proof-of-concept, we had to take a few stories from the upcoming sprint and provide a working example of test cases, executions and plans inside the Jira Xray system. The next step was to analyse test automation cases and recalibrate them from a management perspective.

The current test reporting process, or lack thereof, was both ineffectual and time-consuming. For example, the creation of a test plan for new testing activity would standardly take a couple of hours even if all cases were already described. Our first target was to decrease this to a few minutes.

One of our first jobs was to analyse the needs of test case execution from a management perspective in order to prepare a plan of action. New test executions had to be quick and include the option to edit user scenarios based on given pre-conditions.

We created acceptance tests for each of the assigned proof-of-concept stories and prepared the test management structure in Jira through Xray. Xray allowed us to manage the creation of test plans for upcoming sprints.

We also set up recurring test executions for standard, continuous testing activities like Smoke Testing after product deployment. Feature owners without any QA background were able to test all the applicable scenarios and document through a clear reporting system that generates results throughout the process.

The second stage of our proof-of-concept was to analyse the current status of automated cases and migrate them to Xray. A combination of Cucumber + Selenium and Jira + Xray was easy to integrate because Xray includes native Cucumber test type support and an integrated Automation steps library.

Our target was to create a job in the CI system and configure it in such a way that it would automate taking test cases from test plan, import them as Feature files and send execution reports to Xray. After all existing Feature files were migrated to Xray as test cases, we were able to manage executions in a way that did not require any QA Engineer background.

The integration of Cucumber+Selenium and Jira+Xray led to outstanding improvements in the speed at which new test executions could be prepared and reported on.

Now we were able to create new Feature files for test automation from Xray, avoiding headaches for non-technical colleagues. They were able to create test cases without having to access the Git system at all. All results immediately appeared in Xray without the additional need for results emails or access to the CI system.

Excel-type test plans and reports take too much time to prepare if hundreds of recurring cases need to be executed individually. The new automated test approach meant we could take a test execution made of hundreds of test cases and in a few mouse clicks clone it within the test plan. Including any appropriate default statues and defining test environments.

We achieved:

1. Significant time efficiencies in the preparation step.
2. Clear results demonstrated by statistics-backed reporting for each case.
3. An easy-to-use test automation library.
4. Automated executions managed from Jira’s Xray system in a few clicks without Git/CI system access.

With over 20 years of track record behind us, K&C has become Munich, Germany and the DACH region’s favourite and most trusted outsourced IT consultancy and development agency.

We’re proud to call well-known global brands, established SMEs and exciting start-ups as our partners.

DevOps, Web Development and Cloud Services are our core focuses. Please do get in touch. We’d be delighted to discuss your ongoing needs or a particular project you need outside help with.