{"id":14202,"date":"2020-05-04T06:10:59","date_gmt":"2020-05-04T13:10:59","guid":{"rendered":"https:\/\/blogs.mentor.com\/verificationhorizons\/?p=14202"},"modified":"2020-09-17T12:53:03","modified_gmt":"2020-09-17T16:53:03","slug":"the-ideal-verification-timeline","status":"publish","type":"post","link":"https:\/\/blogs.stage.sw.siemens.com\/verificationhorizons\/2020\/05\/04\/the-ideal-verification-timeline\/","title":{"rendered":"The Ideal Verification Timeline"},"content":{"rendered":"

Our discussion around building integrated verification methodologies started with where <\/i>techniques apply to design by plotting options for verifying low, medium and high-level abstractions<\/a>. That was one step to understanding how techniques fit together in a complete methodology. Today we take the next step with a timeline for when <\/i>all these techniques apply.<\/p>\n

When I started putting this timeline together I intended it to be generally applicable. That only lasted one conversation. Fellow product engineer Matthew Balance immediately pointed out a generally applicable timeline isn\u2019t possible. At a minimum, he suggested, a timeline depends on the scope of the test subject; are we looking at a timeline for subsystem development or an entire SoC?<\/p>\n

Thanks to Matthew, a subsystem timeline is what we\u2019re looking at for now; SoC timeline is TBD. What it shows are all the techniques at our disposal, a recommendation for when each is most effective and the relative impact we should expect. I loosely define impact as value of new results<\/i> (i.e. portion of design verified, number of bugs found, etc).<\/p>\n

\"\"<\/a><\/p>\n

The curve for each technique describes how its impact grows, peaks, then tapers off as the value of new results diminish.<\/p>\n

The pace at which impact builds for a particular technique is proportional to its infrastructure dependencies and\/or the state of the design. As an example for infrastructure, constrained random test has large infrastructure dependencies so its impact grows slowly. Reachability checking, on the other hand, is deployed without infrastructure so it ramps up very quickly. With respect to design state, that may apply to the size of the code base or its quality. The impact of lint, for example, grows slowly at first with the size of the code base then flattens out to incrementally deliver new value every time lint is run. In contrast, constrained random coverage grows slowly as the quality of the design improves and more of the state space can be explored without hitting bugs.<\/p>\n

The relative height of the peaks is subjective. After many <\/i>discussions and tweaks to this graphic, I think the height and duration of each peak pretty accurately captures what we\u2019d see if we averaged the entire industry. That said, I\u2019m certain someone out there will curse me for suggesting the impact of property checking is less than they\u2019ve experienced or take exception to a perceived bias toward constrained random testing. I\u2019ve done my best to capture relative impact while acknowledging case-by-case mileage may vary tremendously.<\/p>\n

Time on the x-axis is split into development stages. Design is the before-any-coding-happens step. It\u2019s not relevant to this discussion yet but I expect it to become relevant when we talk about infrastructure down the road. For now, just consider it as the start line. RTL coding is where the design is built. Sanity marks the first functional baseline for the subsystem where focus is on design bring-up and pipe cleaning features with happy path stimulus. The bug hunting phase is where we start flexing the design beyond the happy path. We expect to find bugs at this point – which is why I call it bug hunting. As the design matures, we transition into the regression phase. Focus here is still on testing; new and longer tests that push the design into darker corners of the state space. Last is closure where focus turns to complete state space coverage through comprehensive stimulus and analysis.<\/p>\n

Two sets of recommendations I intend for people to find in the timeline:<\/p>\n