In-sprint QE automation improves code and accelerates development speed

Speed and quality have always seemed mutually exclusive for engineering development -- but do they have to be? In order to ensure high quality, builds need to be thoroughly tested. However, testing takes time, and leads to slower production and missed deadlines. If teams release a product without implementing sufficient testing, then bugs or crashes follow. Finding a solution to this dilemma has puzzled engineering teams for a while.

One of our large retail customers was having a similar issue with their development. Originally, their development team was not producing quality results. Even when they expanded their quality assurance team, their builds remained bogged down due to errors and the lack of speed in manual testing. The client then invested in automated testing, which improved the quality of the code. However, each test needed to be custom-built, overwhelming the QA team, who could not keep up with this high demand. This created a bottleneck: slowing production to the speed that the QA team could write automated tests.

In order to have regular releases, the customer wanted to achieve in-sprint automation. This is an ideal development scenario where the entire testing process (creation, implementation, execution, reporting) all happens in one sprint. The result of in-sprint automation would be high-quality code with low human involvement, all performing quickly. The client turned to Grid Dynamics' continuous delivery experts to design an end-to-end in-sprint automation. Let's take a look at how we did that.

The high-level solution

Our solution was creating a QE library that could hold reusable tests-- a place to store test scripts. These can then be taken out of the library for other projects, simplifying the test creation process significantly. There were four components to building this solution:

1. Use Behavior Driven Development (BDD) to create functional tests quickly and to simplify the testing process
2. Embed tests from BDD into the CI/CD pipeline, so they will be part of the build process and run for every code commit
3. Create automation for other tests such as regression and continuous performance testing (CPT)
4. Build up a repository of reusable test scripts from large test blocks which are universal for the domain of each project, creating a QE library with tests that can be repurposed for other projects

Now, let’s dive into the details of our solution. We will start with the implementation of Behavior Driven Development as our approach to creating functional tests.

Easier test creation with behavioral-driven development

Behavior Driven Development is an approach that simplifies functional testing. First, the QA analyst writes up the test requirements for a scenario in Gherkin. This is the same as plain English, but simplified to key words and phrases. Each scenario tests one functionality of the code. The first time these scenarios are entered into Java, the QE engineer must define them manually. For every subsequent use, the Gherkin scenario is translated to Java test steps automatically. Once the tests have run, reports are created with the results and sent to the QA analysts for review. Here’s an infographic that details the BDD process with an example:

BDD simplifies the testing process in two ways. First, requirements would often get lost in translation between the explanation from the analyst to the engineer, creating tests that didn’t quite fit the requirements. With BDD, the requirements are directly inputted, so the tests cover exactly what is needed. Second, test reports used to be created by business system analysts, who would then send them to the QA team. These reports are now self-documented, so a person does not have to write them manually. This further reduces human involvement and speeds up the release process.

Tests were embedded into the CI/CD pipeline

To automate testing, we had to integrate the testing process with the CI/CD pipeline, with the goal to have builds run every night. After the functional tests were configured for use in Java as part of the BDD (as written above), they were embedded into the CI/CD pipeline. When the embeds activated, they created a testing environment, ran the tests, and then destructed, saving resources. As mentioned above, the test results were then documented in a report, and sent to the QA analysts for review. The tests were now part of the regular build pipeline, eliminating manual input. This integration saved time and removed the potential for human error in executing tests.

We also automated non-functional tests and increased coverage of metrics

In order to speed up the development process, we added additional tests to the release pipeline that used to be executed manually. This extended the automation process to regression and continuous performance tests in addition to functional tests. Similarly to the functional testing, once these other tests have been executed, results are then sent to the QA team, who sign off on the tests. Adding these to the nightly builds kept the client constantly up-to-date on the performance of their systems, which improved their code and sped up the process.

We also wanted to ensure that changes being made were positively impacting the client’s performance. This is where automating continuous performance testing helped. By adding this to the pipeline, the developers were able to see how their changes impacted performance immediately. Therefore, if a change they made was negative, it could be quickly adjusted. Additionally, dashboards were attached to projects to track KPI’s (key performance indicators), enabling overall observance of how changes effected important metrics. Finally, open source tools were introduced in place of licensed ones, further cutting down on costs.

Building the QE library

The final step in the solution is the accumulation of reusable test cases in our code repositories. These tests are naturally stored once they've been created, forming a library of QE tests. Each "book" in the library is the compilation of tests for each specific project. All of the tests created through BDD are designed to be reusable -- they are script archetypes from bigger, universal test blocks that apply to more than just one project. Since each project repository is kept separate, test access and retrieval are easy. This library is helpful in two ways -- it can help test creation within a single project, and can also aid in inter-project reuse. As long as the domains of the projects are similar, test cases can be taken from one project and moved to another to be reused with only minor changes. As more projects are undertaken with the client, and the corresponding number of tests increases, the number of books in the library grows as well.

Results

By replacing manual processes with automation, we improved testing quality and sped up the development process. We helped the client achieve in-sprint automation: test development, execution, and reporting all happen in one sprint now, meaning that the client is no longer constantly behind in production. Meanwhile, less necessity for a large QA team and the replacement of licensed tools with open source software helped reduce costs.

The presence of the library also assures that time spent creating tests will be reduced in the future. The tests created are all reusable -- they apply to more than one specific testing scenario, and can be used for other projects. The presence of previously created use cases can speed up development by as much as three days. And, as the number of tests in the library continues to grow, more and more time will be saved. Additionally, we hope to implement automated load and stress testing soon, further reducing time spent on manual testing.

The hope will be that at some point, almost the entire process will be automated. Tests will be taken out of the library when needed for a project and integrated into the project’s pipeline, which will then run by itself when code is committed. Testing will then become quick and simple, as there won't be human involvement causing delays or errors. In-sprint automation with our QE library and a test automation framework is the answer to the engineering paradox: you have achieved both speed and quality.

Conclusion

QA testing has long been a bottleneck for production. Companies must invest in QA so they can thoroughly test code before release, but the time needed to create and execute tests slows production significantly. However, new methods of automation can cut down on human involvement. Grid Dynamics’ QE Library is a breakthrough for QA automation, enabling much of the process to be completed without human input. This won’t eliminate errors from testing, or guarantee on-time production, but it will make testing delivery much speedier, helping to achieve in-sprint automation. This solution will work for everyone, as integrating tests into a pipeline and building a library for reusable test scripts is not specific to any one company or project.

Automation is the way of the future with QA testing. We have learned many things so far in our work on implementing automated test creation and execution, and are always looking for new challenges. If you are a retailer that wants to automate your testing process, please give us a call!