Product quality: belief or proof?

There are two aspects of any product that give me great pleasure: good design and quality of manufacture. This applies to just about anything: a pair of shoes, a car, a house , a piece of software … Assessing both these parameters can be hard. A pair of shoes must look good, be comfortable to wear and last well; this takes time to evaluate. A car needs to be enjoyable to ride in, drive well and offer long term reliability and economy; again, not something to be measured quickly. A house can be even more difficult, as you need to live in it through a full year of seasons at least and, even then, you may still be evolving the way that you use the space, which will establish how good the original design was.

In many ways, software can be the most difficult product of all to assess, as programs represent the most complex “machines” mankind has ever produced …

To explain this last point, as I mentioned on a previous occasion, the most complex mechanical device ever created is the Space Shuttle orbiter, which has around a million moving parts. In software terms, I suppose a bit is a moving part and program code consisting of 100m bits is not unusual. It is, therefore, unsurprising that assessing the design of a program and figuring out how well it was implemented is a big challenge.

How well a program is designed can often be determined by just using it. Fortunately many software vendors offer evaluations to enable potential users to try before they buy. On countless occasions, I have downloaded and installed a promising program, only to uninstall it after only a few minutes use, during which I concluded it was not for me.

Quality is more difficult. To some extent, you need to rely upon the vendors claims and reputation. For embedded software products, the possibilities for quality control and detailed testing are quite variable:

• An assembler has a very specific range of functionality and a testing procedure may be developed that exercises it to almost 100%. Macro programming is the only area where testing is necessarily limited.
• Linkers are more flexible, but testing can stretch them in many ways, but there remains some possibility for unforeseen “corner cases”.
• A compiler is another matter. There are numerous test suites available, but there are an infinite number of possible programs that may be written and, hence, the same number of opportunities for finding a flaw in a compiler.
• The kernel of a real-time operating system [RTOS] can have its basic functionality tested to something like 100%, but there is also stress testing and performance measurement to consider.
• Other components of an RTOS, like networking, are readily testable in well recognized ways. This is a topic I covered previously.

The ways that vendors portray quality in embedded software products is interesting. Some companies use beta testing programs to get their customers to find bugs. This is not a recommended way to build a lasting good reputation.

Other companies are more creative. One of the first RTOS products on the market was VRTX from Hunter & Ready [later Ready Systems, who were acquired by Microtec Research, who were acquired by Mentor Graphics …]. They were very confident about their product being bug free. They ran an advertising campaign called “A Bug for a Bug”, where they offered a VW Beetle car as a reward for finding a bug in the kernel. I have no idea whether there were lots of strings attached or if they ever gave away a car, but it was a very memorable program.

When you are buying a software product, do not be shy about querying the quality assurance and testing procedures in use. A salesman saying that he believes in his product is good, but not good enough; I want proof!

Years back, there was a compiler vendor [I will not name them – they were later acquired by a semiconductor company, so never heard from again] who offered quite good quality documentation. At the bottom of many pages were Latin phrases praising the glory of God. I have total respect for other people’s religious beliefs, even if I do not share them. But this is not the business of software manuals. We always used to say “They have faith; we have Technical Support.”