Clearing the Fog of ISO 26262 Tool Qualification

Introduction

Developing products to the ISO 26262 standard requires many activities across multiple disciplines. One of those activities is ensuring the tools deployed within the development are suitable for activities in the project. When we say suitable, we need to be confident that if a tool malfunctions, that failure will be caught before violating a safety requirement. Unfortunately, tool qualification isn’t clear cut, and there is a lot of misunderstanding and, frankly, misinformation around the qualification of software tools. The remainder of this blog post will help provide insight into some of the issues Siemens safety experts see in the industry today and the little-hidden truths and pitfalls which could lead to schedule delays and additional programs costs. We’ll try to avoid the scary depths of tool qualification, but it is essential to define some key concepts. 

Background

ISO 26262 dictates that tools used in the development of safety-critical electronics be assessed, and it guides this analysis by defining three terms:

ISO 26262 Term	Definition
Tool Impact (TI):	This is an evaluation of the potential of a tool to introduce or fail to detect errors. The standard defines: TI1:  Selected when there is an argument that there is no such possibility. TI2: All other cases. Practically speaking, many EDA tool use cases will be TI2.
Tool Detection (TD)	This is an evaluation of the confidence that the tool will prevent the malfunctioning of a software tool, and the standard defines: TD1:  High confidence that a malfunction will be prevented or detected. TD2: Medium confidence that a malfunction will be prevented or detected. TD3:  All other cases
Tool Confidence Level (TCL)	An establishment of tool confidence based on TI and TD and the standard defines: TCL1:  High level of confidence.  No further qualification methods are required. TCL2:  Medium level of confidence. Further qualification methods are required. TCL3: Low level of confidence.  Further qualification methods are required.

The classification of a tool starts by first deciding on TI and TD for the tool under analysis. The setting of TI and TD ultimately determines the TCL for the tool based on the table below, found in ISO 26262-8:2018 Clause 11.4.5. It is important to note that TCL determination is not based only on the tool but also on its use in the context of the process step for the end product.

FURTHER QUALIFICATION METHODS ARE REQUIRED when TCL is determined to be TCL2 or TCL3.  The ISO 26262 standard provides four methods for performing tool qualification (ISO 26262-8:2018 Clause 11.4.6). The standard also indicates the suitability of each method based on the ASIL target.

The determination of TCL is required for software tools purchased from an EDA vendor and tools developed internally to support the development of the product.

Clearing the fog

Now that we have covered some of the fundamental concepts, let’s discuss areas where misconceptions and misinformation often lead to confusion among practitioners.

Statements regarding the “Achievement of TCL#”

You may come across language that states a tool has achieved TCL# certification (i.e., Achieved TCL2 certification). This is confusing and often misleading because TCL is not something that is achieved. TCL is a classification of the tool by its function performed in the context of the project activities. As shown above, this classification is based on TI and TD. During a tool suitability assessment, the tool is classified, and subsequent qualified. It is this classification and qualification method that is certified, as previously shown in Figure-X above.

Best Practice: When evaluating a purchased software tool, focus on the qualification method and determine if the method is a highly recommended technique for your ASIL target. If no method is applied because the tool was evaluated as TCL1, review the rationale behind the TI, TD, and TCL and confirm the rationale conforms to your development process.

TCL# is better than TCL#

Often, you’ll see claims that TCL2 is better than TCL1. The common misconception is that because TCL2 requires additional rigor, TCL2 is considered safer than TCL1. It is important to remember that the setting of TCL is simply based on a classification of the tool, established through the evaluation of TI and TD. TI2 is a typical result in EDA tools because of their usage in the development lifecycle. Synthesizers, Simulators, BIST Insertion, Physical placement, verification, etc… are all TI2 because of the impact they could potentially have on the end product. TD is subjective, and we believe it’s challenging for an EDA provider to claim TD1 without providing sufficient rationale and evidence supporting this claim.

Best Practice: Don’t believe one TCL is better than another TCL. When evaluating a purchased software tool, identify the TCL of the tool and the rationale behind the TI and TD classification match your tool use case. If a tool is declared TCL2/3, pay close attention to the qualification method deployed to ensure that method is suitable for your ASIL target.

Certification of future versions

Often hidden in the text of qualification reports are claims that current tool versions and all future versions are covered under the certification. Logically, this is a falsehood. Ask yourself, is there some future product one can claim is certified because one certified a past variant? Software tools are no different than any other product in that they change and evolve. This includes new use cases, new features, new processes, and changes to other aspects of tool development (issue management, regression systems, etc…). Minimally, these changes require a re-classification of the rationale supporting the qualification method, and that classification may result in the need to recertify.

Best Practice: When evaluating 3rd party vendor reports, pay close attention to claims of future software versions covered.

The hidden cost of ownership

Are you assuming responsibility for all of the tools in your toolchain? Does the qualification report you receive from your vendor provide the maximum value, or are there disguised risks? Does your vendor take ownership and responsibility for the validation evidence? All of these are great questions to ask! There are two hidden costs to mention here.

The qualification method

As stated earlier, the standard defines four methods for tool qualification. Tool vendors providing certifications will choose which methods to adhere to. Specific methods aren’t suitable for the higher ASIL targets. Note that in ISO 26262:2018, method ‘1d’ is no longer highly recommended for tools classified as TCL2.

Best Practice: Review which method was followed and whether that method is highly recommended based on your ASIL target. This would also be an excellent time to evaluate future projects and whether there is an expected evolution to a higher ASIL in a future product variant (i.e., ASIL-B -> ASIL-D). If ASIL-D is on the horizon, it makes sense to ensure your design flow is future-proofed.

Who is accepting liability?

Simply put, not all tool vendors are equal. Some vendors will add disclaimer clauses stating the product team is responsible for confirming all evidence, and communicating the evidence is not the vendor’s responsibility.

Best Practice: Review the certification work products for disclaimers that absolve the vendor of responsibility or statements passing the burden of validating the evidence. If your vendor evades responsibility, understand the impact this may have on your team if re-validation of your design flow and toolchain is required. Additionally, verify the vendor is maintaining and keeping this evidence for 20 years past the release date.

The nuances in the certification of flows

Often, EDA vendors will certify tools in a prescribed design flow and toolchain. Deploying design flows as a means of independent assessment is a widely used and acceptable tactic to validate tool behavior. By deploying checks and balances in the design flow, one can reasonably set the TCL as TCL1, thereby avoiding any need for tool qualification.

Before accepting this wholesale, it is important to ask a couple of questions:

1. Does the certified design flow align with your design flow?
2. Do the tools deployed in the certified flow align with the tools deployed in your design flow?

If the answer is no, it’s essential to understand the impact, for even the slightest difference will void the certification. This will result in additional cost in re-validating the use cases to your design flow.

Best Practice: When evaluating 3rd party vendor reports, pay close attention to requirements around the usage of specific tools in a prescribed flow. When flows are used as part of a certification, look for flows that allow flexibility in the tools used and their configurations.

The Siemens Approach to Tool Qualification

After careful analysis, we concluded that ‘1c’ is the best approach as it delivers the most value to our customers. 

Why ‘1c’?

When evaluating which method to pursue and have certified, we concluded that ‘1d’ provides an indirect method of providing confidence. What I mean is, if a process is in place and that process is audited to ensure it complies with a standard, there is no guarantee a process was followed. Earlier in my career as a Quality Engineer, the number one reason for defects in a product was because people did not follow the process. If people do not always follow the process, how can that process provide the highest confidence in a tool?  In the end, “1c” provides a direct method for providing confidence in a software tool and is also “highly recommended” for the most stringent ASIL targets.

Other key points:

1. Siemens qualifies its tools regularly as new software versions are released through validation (Method 1C). This ensures we continuously evaluate and verify the software tool as new features and use cases are introduced.
2. Siemens explicitly states that it accepts responsibility for all validation evidence detailed in the reports.
3. Siemens provides the certifications and report for usage in your broader safety case

Conclusion

Tool qualification is often murky, with a lot of information and claims being made. Make sure you understand your design tool flow and how 3^rd party certifications apply. Unfortunately, that may require you to bring out your inner Sherlock Holmes. Hopefully, the information provided here gives you a couple of key things to look out for when tackling tool qualification. Please reach out if you’d like to learn more about the Siemens approach and the collateral we provide.

Other Topics

This post is part of a broader safety series highlighting the challenges practitioners face during the development of safety critical ICs. Please refer to Guidelines to a successful ISO 26262 Lifecycle to view other posts in the series.