How to Maximize Automation ROI By Prioritizing Testing Scope?

Automation testing scope identification and prioritization

There is a common saying that ‘you cannot improve what you cannot measure’, this stands true for test automation as well. I have seen teams leaping to start automation without defining & prioritizing the scope of automation testing. Without defining and planning automation, there is no way we can measure the automation’s success and its ROI. We end up automating low-value test cases that should not be automated in the first place.

To avoid such a situation, defining the automation testing scope & prioritizing it should be the starting point of test automation. This will act as the foundation for every decision or activity that will happen as part of automation implementation.

How to Prioritize Automation Scope to maximize ROI?

I would like to introduce a simple yet effective framework that will give us an efficient mechanism to identify -

1. Automation Scope — if the test case should be automated or not?

2. Priority Order — In what order test cases should be picked for automation?

The following 2x2 matrix will help in defining these two factors. We can implement this in our automation projects by categorizing our test cases on this matrix. The matrix has the following axis to measure test cases’ -

1. ‘Business Criticality’ — x-axis.

2. ‘Repetitiveness’ of execution frequency throughout the lifecycle of the software — y-axis.

Framework for Automation testing scope identification and prioritization

We need to categorize our test cases based on two factors (Business Criticality and execution frequency) and select which quadrant of the above matrix the test case will belong to. To do this, for each test case, ask these two questions -

1. What is the Business Criticality of the test case?

2. What is the execution frequency of the test case throughout the life cycle of the product?

We need to answer these questions with two options — either ‘High’ or ‘Low’.

Based on the answers to these two questions, where the answer should be only in ‘High’ or ‘Low’, categorize the test cases in one of the four quadrants of the above 2x2 matrix.

We would eventually have four sets of test cases placed in four quadrants as follows —

Quadrant 1 — High Business-critical and High repetitiveness — The test cases falling in this quadrant will give us the highest positive automation ROI and will give us the most benefits as test cases are business-critical.

Quadrant 2 — Low Business-critical and High repetitiveness — The test cases falling in this quadrant will give us a positive automation ROI and the business impact of automation will remain low.

Quadrant 3 — High Business-critical and low repetitiveness — The test cases falling in this quadrant will give us a low automation ROI and can be kept as manual regression test cases or should be done after completion of the above two cases.

Quadrant 4 — Low Business-critical and Low repetitiveness — The test cases falling in this quadrant will give us negative ROI and should not be in automation or even manual regression pack.

This is a simple and powerful way to manage manual and automation test suites. Let’s summarize the framework -

1. The first two-quadrant test cases should form the bulk of the automation scope and is the GREEN zone for automation with positive automation.

2. The third quadrant test cases form the manual regression pack, and should be considered for the manual regression pack as automation return will be low and we call it the AMBER zone for automation.

3. The fourth quadrant test cases should not be part of manual regression or automation and this quadrant is the RED zone for automation.

Conclusion

Test automation brings many benefits like consistent results, removing human dependency, efficient use of the team members’ time by letting them focus on more productive tasks, etc. However, the main objective of automation is to get a return on investment (ROI). An unplanned approach will defeat the purpose of automation. Hence knowing what to automation along will the priority order will maximize the ROI potential of the test automation.

Learn - Common mistakes test teams make in automation testing and how to fix them