How to certify a compiler for functional safety?
I work in the toolchain team at a company with ISO-26262 certification, and we recently had our development suite validated for functional safety.
Part of the process was running a set of validation test suites in which thousands of tests programs were compiled and the results compared against expected results. Another part was ISO standard-conformance tests. None of these tests, of course, are exhaustive, but did identify some issues. Yet a third part consisted of running the DejaGNU test suite that comes with GCC itself.
The next trick for safety validation is to make sure all known issues are documented. Functional safety does not mean your toolchain is perfect, it only means the known imperfections are clearly documented and that you have a process in place to identify and document imperfections. What you need to do for full validation is to fix or document and justify each and every deviation from expected behaviour so there are no known, unjustified deviations.
Validation is an entire industry on its own. It is expensive and time-consuming.
start-disclaimer: I am part of a team that develops a test-suite to validate C/C++ compilers and qualifies compilers for functional safety. end-disclaimer
It is possible. The process is called ‘qualification’ not certification because it is aimed to find any weak spots in the compiler and define workarounds if needed. In ISO 26262 it can be found in Part 8, Section 11 “Confidence in the use of software tools”. The “software tool” in this case is the compiler.
Section 11.4.9.2 says:
11.4.9.2 The validation of the software tool shall meet the following criteria:
the validation measures shall demonstrate that the software tool complies with
its specified requirements,
...
EXAMPLE
The standard for a programming language helps to define the requirements for
validating the associated compiler.
explaining that must be coherent with the ISO standards. To do the verification, you need a test suite that is based on the language standard.
The DejaGNU suite is not suitable to qualify a compiler for functional safety. DejaGNU is useful to recognise if there are some "well-known-issues" in your compiler version, but it does not systematically validate the compiler against any ISO standards. It is mostly a regression test suite, and tests many requirements that are not in the ISO standards. Here are some examples of that:
eg 1: https://github.com/gcc-mirror/gcc/blob/master/gcc/testsuite/c-c%2B%2B-common/diagnostic-format-json-1.c
This test tests whether the compiler produces diagnostic messages in JSON format, which is not required by the ISO/IEC-9899:* C standard, ISO/IEC-14882:* C++ standard or the ISO-26262 standard.
eg 2: https://github.com/gcc-mirror/gcc/blob/master/gcc/testsuite/g%2B%2B.dg/tree-ssa/pr13954.C
This test "fails" if your compiler does not implement any kind of conditional-constant-propagation optimization, but this is not required by the C++ standard neither by any functional safety standard.
eg 3: on the opposite side, passing the DejaGNU suite does not indicate any compliance to the standard, but only with the GNU "dialect" :
https://cpp.godbolt.org/z/Gyu_i5
It is also important that the compiler is verified for the options, configuration and environment with which it is used in application development. This is commonly called the 'use case'. ISO 26262 says:
11.4.3.1
When using a software tool, it shall be ensured that its usage, its
determined environmental and functional constraints and its general
operating conditions comply with its evaluation criteria or its
qualification.
As mentioned correctly by @stephen m. webb testing is one part of the certification process. The other part is to document the process, the test results and the mitigations (workarounds).
After you validate your compiler you can reuse it in any safety-critical environment, as long as the use case is the same.