Regression Automation is all over the map today. The software testing world is way more varied than people think. Some organizations have thousands of tests constantly running on all kinds of VMs taking multiple days to execute, while others have no test automation at all. Any hope of Continuous Delivery or Pipeline Automation makes this current state unsustainable.
The ability to “automate everything” is a myth. The understanding that many teams have of regression today only a fuzzy confidence or consistency but not necessarily improved quality.
Many teams with significant regression automation are unable to reach confident results due to the number of failing tests, and a history of false positives and negatives. These teams must rethink WHAT they automate and WHEN to run the tests.
Test automation must now reflect deployments defined at different states of readiness and environments. Teams must evaluate the time spent running automation over and over, as well as the time to analyze the results. They must also consider the time required to maintain automation. And most importantly, they must realize that if a bug was missed the first time the suite was run, that bug will consistently be missed.
We must rethink automation to support a streamlined continuous delivery pipeline. If you have no automation, begin now. Teams must move from being “bloated and slow” to “lean and mean” to build suites with tests that have a specific purpose and outcome.
