Thought Leadership

Conclusion: The 2018 Wilson Research Group Functional Verification Study

March 19, 2019 • 4 MIN READ
By Harry Foster

Deeper Dive into Non-Trivial Bug Escapes and Safety Critical Designs

This blog is a continuation of a series of blogs related to the 2018 Wilson Research Group Functional Verification Study (click here).  In my previous blog (click here), I presented verification results findings in terms of schedules, number of required spins, and classification of functional bugs. In this blog, I conclude the series by having a little fun with some of the findings from our recent study.

You might recall from our 2014 study we did a deeper dive into the findings made a non-intuitive observation related to design size and first silicon success. That is, the smaller the design the less likelihood of achieving first silicon success (see 2014 conclusion blog for details). This observation still held true in 2016 as well as this year’s study. This is likely due to projects working on smaller designs typically are less mature in their verification processes, as well as a higher percentage of designs with analog.

For this year’s study, we decided to do a deeper dive related to the following:

  1. Verification maturity and non-trivial bug escapes into production, and
  2. Safety critical designs and silicon success.

Verification Maturity and Silicon Success

The study results show that the IC/ASIC market has matured its verification processes overtime, and we see the FPGA market also starting to mature its processes. This maturity is likely due to the growing complexity of designs and related efforts to control cost and engineering headcount through the adoption of FPGA design and verification solutions that increase productivity.

Perhaps the most concerning finding from this year’s study relates to the number of FPGA projects with non-trivial bug escapes into production. However, we did find an interesting correlation between the improvement of reduced functional flaws contributing to non-trivial bug escapes and the maturing of FPGA projects’ functional verification processes. The data suggest that projects that are more mature in their functional verification processes will likely experience fewer bug escapes.

To test this claim, we partitioned the study participants into two separate groups: FPGA projects with no bug escapes and FPGA projects that experienced a bug escape. We then examined the percentage adoption of various verification techniques. The results are shown in Fig. Conclusion-1. You might note that the  various verification techniques related to bug escapes and no bug escapes does not sum to 100 percent.  The reason for this is that we performed our analysis in terms of technique adoption independently on the separate groups: bug escapes and no bug escapes.

Figure Conclusion-1. Process Maturity and Non-Trivial Bug Escapes

These findings are statistically significant in that the group with no bug escapes tended to have higher adoption of various verification techniques, which suggest they are more mature in their verification process. However, what we are unable to measure from our study is how effective a project was in adopting any of these processes. For example, a project that experienced a bug escape could claim that they have adopted functional coverage, yet the fidelity of their functional coverage model might be poor due to their inexperience. From our study data, we are unable to assess successful or effective adoption for any particular verification technique.

Safety Critical Designs and Silicon Success

The second aspect of our 2018 study that we decided to examine a little deeper relates to safety critical designs and their silicon success. Intuitively, one might think that the rigid and structured process required to adhere to one of the safety critical development processes (such as, DO-254 for mil/aero, ISO 26262 for automotive, IEC 60601 for medical, etc.) would yield higher quality in terms of preventing bugs and achieving silicon success.

Figure Conclusion-2 shows the percentage of FPGA projects that claimed to be working on a safety critical design.

Figure Conclusion-2. Percentage of FPGA projects working on safety critical designs

We partitioned the findings into two groups: projects working on non-safety critical and safety critical designs. Then in Figure Conclusion-3 we examined the number percent of bug escapes and no bug escapes for projects working on safety critical designs.

Figure Conclusion-3. Non-trivial bug escapes for safety critical vs. non-safety critical FPGA designs

Clearly, the data suggest that a development process adopted to ensure safety does not necessarily ensure quality. Perhaps this is non-intuitive. However, to be fair, many of the safety critical features implemented in today’s designs are quite complex and increase the verification burden.

This concludes the findings from the 2018 Wilson Research Group Study.

Quick links to the 2018 Wilson Research Group Study results

Harry Foster

Harry Foster is Chief Scientist Verification for the Design Verification Technology Division of Mentor, A Siemens Business; and is the Co-Founder and Executive Editor for the Verification Academy. In addition, Harry is serving as the 2021 Design Automation Conference General Chair. He holds multiple patents in verification and has co-authored six books on verification. Harry was the 2006 recipient of the Accellera Technical Excellence Award for his contributions to developing industry standards, and was the original creator of the Accellera Open Verification Library (OVL) standard.

More from this author

What to read next:

Leave a Reply

This article first appeared on the Siemens Digital Industries Software blog at https://blogs.stage.sw.siemens.com/verificationhorizons/2019/03/19/conclusion-the-2018-wilson-research-group-functional-verification-study/