Prevent systematic faults by adopting third-party test tools now!
Model based development (MBD) is seeing greater adoption rates among embedded software teams to simplify the production of complex applications. By abstracting software architecture, design, and behavior into commonly understood visual representations (the model), MBD helps developers collaborate and maintain software more effectively while identifying potential system issues earlier in the lifecycle.
As many MBD tools support automatic code generation and testing, embedded software teams also see increased productivity. The challenge for safety-critical systems, however, lies in the verification and validation of auto-generated artifacts in environments with strict functional safety requirements. Many software standards require source-code level testing and compliance and some mandate validation down to the object code.
To reduce the uncertainty around testing MBD-driven code artifacts, development teams must go beyond the capabilities of the MBD platform to leverage independent and automated test built to support functional safety certification.
Model based development and functional safety MBD enables developers to specify software components, relationships, and behavior using well-defined syntax and semantics. A popular example is the Unified Modelling Language (UML), a set of notations for real-world software elements like classes, objects, and interfaces that are assembled and connected onto diagrams that represent the architectural and behavioral aspects of the system.
MBD tools generally include automatic code generation based on the visual model. The code generation process usually involves a transformation from the visual model to an intermediate implementation model that accounts for language, compiler, and other environment-specific characteristics. This model generates the source code that is fed into the compiler and linker to create the object code and executables.
For systems concerned with functional safety, such as automotive, medical, and aerospace products, these transformations from model to object code raises two questions: “How to detect and prevent potential issues introduced by the MBD tool?” and “Does the application still meet requirements?” This is why ISO 26262, “Road vehicles—Functional safety,” and the aviation standard DO-178C (through the DO-331 supplement) include sections for MBD.
MBD and ISO 26262
The ISO 26262 standard calls out the potential for MBD to introduce unforeseen issues in application code:
“The potential benefits of this [MBD] approach (e.g., continuity information sharing across the software life cycle, consistency) are appealing but this approach may also introduce issues causing systematic faults.” – §B.2.3, “Potential impact of MBD on the software lifecycle”, ISO 26262
ISO 26262-6 §9.4.2 further states that development teams must provide evidence to show the compliance of source code to its design specification, including cases where developers use MBD.
MBD and DO-178C
In civil aerospace, DO-178C is the primary standard used by certification authorities to approve commercial software-based systems and DO-331, “Model-Based Development and Verification Supplement to DO-178C and DO-278A” covers MBD specifically.
To meet the objectives of the highest design assurance level (DAL) specified in DO-178C, DAL A, a developer must verify that any differences the compiler introduces between source code and object code are valid, justified, and documented. This traceability ensures that every line of code is there for a reason and puts a magnifying glass on the outputs of code generators and compilers.
A question on compilers
Developers use compiler code optimization features to improve the performance of their systems, but they typically avoid these optimizations for safety-critical development. As these features reorganize the order of instructions, data, and memory, they often eliminate instructions and defensive code on the assumption they’re not needed. The potential differences between source code and object code present challenges for any developer seeking certification, as DO-178C specifically requires they be identified and validated. An independent verification and validation mechanism must be employed to reduce the risk of missing systematic faults introduced by code generation or a compiler.
Improving testing with third-party tools
Third-party tools bring “a fresh pair of eyes” to verification and validation that is independent of any mechanism in the code and build toolchain. Acting as a complement to the existing developer tool stack, they are a necessary feature of safety-critical processes seeking to achieve compliance and improve code quality. These tools reduce functional safety risks in a couple ways:
Supplementing SIL and PIL testing:
Software in the loop (SIL) and processor in the loop (PIL) testing have long been essential components of verification and validation programs. SIL testing operates on a model or simulated environment while PIL testing integrates physical devices into the mix. While their intent is to ensure correct functionality of the system under different conditions, it’s difficult to prove code coverage for complex applications without specialized tools. Third-party test tools ease this burden by performing code coverage tests on the source code and object code, with some tools auto-generating stubbing functions to exercise a range of inputs and outputs.
Performing control and data coupling analysis: Control and data coupling is one of the structural coverage types found in DO-178C and similar standards, used to measure the reliability of software architecture based on the interdependencies between its components.
Control coupling analysis determines the degree to which a software component influences the execution of another component (e.g., an interrupt handler that calls a supporting function). Data coupling analysis determines the dependence of a software component on data not exclusively under the control of that component (e.g., two functions that share global and local data through passed parameters)
An independent test tool can perform control and data coupling analysis on both auto-generated and handwritten code.
Reviewing auto-generated code
The report illustrated in the figure identifies several violations, with the example in the inset image showing the inclusion of an if() statement that is always true in the auto-generated code, meaning there is some infeasible code.
Prevent systematic faults by adopting third-party test tools now!
Model based development tools do a good job of stemming the pressure of complex embedded software development, but they pose significant risks for safety-critical systems. Ensuring the model to code transformation process doesn’t introduce systematic failures, or the compiler introducing undesirable behavior in object code, requires independent test tools that can analyze both auto-generated and handwritten code.
Such tools act as a supplement to techniques common in the verification and validation of safety-critical systems, providing a sanity check that improves the automated testing process and reduces the risk of faults entering the field.