Methodology by Example – 6 Approaches to Verification
This blog is an exciting next step – exciting for me at least! – that builds on what I proposed in The Ideal Verification Timeline. In that we discussed a timeline for a complete collection of verification techniques. We looked at when each technique is best applied during the project development cycle and its relative impact. I received some pretty good feedback from that. Very encouraging. Got me thinking that we’re going in the right direction with this string of ideas.
The ideal timeline did feel like a big step. But despite the positive feedback, to me it fell short of being realistic. It includes all the possibilities and for most development teams all the possibilities aren’t practical.
This post is meant to take us into the practical.
We do it with a collection of example verification flows pulled out of the ideal timeline. Each example is part of a progression that builds up from an exclusively constrained random flow. At each step in the progression, I add one or more techniques and estimate the consequences; consequences are measured in bug rate and confidence. This progression wasn’t originally part of my plan; luckily it sprouted from a comment from a colleague on an earlier post (I take that as proof there’s benefits to constantly pestering colleagues for opinions 🙂 ).
I went with video this time because I felt like it was more appropriate for the visuals. It’s about 7:20 long. As always, happy to have questions or comments. You can find me at: neil_johnson@mentor.com.
-neil
Comments
Leave a Reply
You must be logged in to post a comment.
Hi Neil,
Interesting observations & very systematic collection of data. A quick tangential question –
With all the additional techniques getting added to aid in improving the confidence, I am wondering that the area of the blue mountains in the graph should shrink either in impact or on timeline with deployment of each additional techniques OR are we suggesting that the additional techniques are completely disjoint to the impact that Directed/Constrained Random Verification will have!
gaurav, in a nutshell, yes… but :). Adding a technique to a flow will change what’s required from the others, no doubt. so in terms of what’s accomplished, to follow your example, less time would be spent doing constrained random testing as other techniques are added b/c those other techniques would have already exposed issues that would no longer be found in cr tests. everything is definitely connected.
the reason I didn’t change the sizes/duration on the impact plot is b/c I mean for that to still imply best case timing and relative impact. If we plotted effort here, that would be different and would obviously change as we go through this progression. the length of the timeline would also (hopefully) change, the lengths of each segment, etc. realizing this is very dynamic, I tried to keep it simple by sticking to a recommendations about where techniques are best applied with a feel for how impactful they can be relative to each other. hope that helps!
-neil
Thanks for clarifying Neil. Makes sense!