Revolutionizing software testing: Introducing TCP-Net++
By Mohamed Abdelkarim and Reem El Adawi
In the dynamic world of software development, balancing speedy delivery with quality assurance is akin to walking a tightrope. Enter TCP-Net++, a groundbreaking tool in the realm of test case prioritization (TCP) developed at Siemens EDA. Building on our initial product, TCP-Net, the enhanced TCP-Net++ tool takes innovation to the next level, demonstrating impressive results in real-life industrial environments and providing significant improvements in efficiency and performance. We discussed the evolution of TCP-Net and presented our deployment results and findings at the 2023 IEEE International Conference On Artificial Intelligence Testing (AITest).
TCP-Net: The foundation
Our journey began with TCP-Net, a novel tool that combined coverage and history features in a deep neural network to effectively prioritize test cases. Designed to address bottlenecks in regression testing, the TCP-Net tool revolutionized this process. The growing number of test cases, coupled with the adoption of continuous integration, had made it challenging for QA teams to run all regression tests efficiently. TCP-Net tackled this issue by learning a high-dimensional correlation between source files and test cases, leading to significant reductions in time and resources without compromising quality. More information regarding TCP-Net is available in our original paper.
The evolution to TCP-Net++
Building on this solid foundation, TCP-Net++ incorporates end-to-end deep neural networks and ensemble learning techniques. These enhancements boost performance and ensure stability over time. TCP-Net++ can smartly identify which test cases are most critical and must be run first, catching potential issues early. This approach not only saves valuable time but also significantly reduces hardware resource requirements.
Deploying QA software in the real world
Adapting TCP-Net++ for deployment in real-world scenarios was challenging. We encountered and overcame various roadblocks, fine-tuning the tool to enhance its performance and usability. A key enhancement was the inclusion of ensemble learning, which is a powerful technique in machine learning that combines the predictions of multiple models to improve the overall accuracy and robustness of predictions. In ensemble learning, multiple models are trained on the same data using different algorithms or parameters, and their predictions are combined using various methods such as averaging, voting, or stacking. The principle is that a group of diverse models can make more accurate and stable predictions than a single model alone.
Productivity impact of AI-based QA
Our deployment analysis reveals that TCP-Net++ can achieve a staggering 90% to 100% failure coverage by running just 23% to 39% of the test cases. This efficiency is a game-changer, leading to quicker software releases without compromising on quality.
Why faster, smarter QA matters
In today’s fast-paced tech landscape, efficiency and reliability in software testing are more crucial than ever. TCP-Net++ addresses this pressing need using its ability to prioritize test cases accurately and quickly to facilitate faster software releases without compromising quality. Companies that produce software products rely in large part on the reputation of their software for market success. Ensuring each new version release maintains the expected high quality is essential.
Looking Ahead
Our journey with TCP-Net++ doesn’t stop here. We are committed to continually refining the tool, exploring the impact of different hyperparameters, and evaluating its effectiveness across a broader range of industrial environments. Our goal is to ensure that TCP-Net++ not only meets but also exceeds the evolving demands of software testing.
For those interested in delving deeper into the technical intricacies and the full potential of TCP-Net++, we invite you to explore both papers for more details. Join us in exploring the future of efficient software testing!
TCP-Net: Test Case Prioritization using End-to-End Deep Neural Networks
