Changing gear from focus on quality of miles to quality of coverage in ADAS and AV testing

Ensuring safety is one of the biggest challenges in the development of self-driving vehicles. Many experts are concerned that safety methods and metrics based on quantity of miles driven in simulation and road testing, the number of disengagements, and/or traditional test coverage are insufficient, non-scalable, and not easily shared or reused.

In addition, due to the autonomous, uncontrollable behavior of automated driving systems and traffic, developers cannot be sure their tests are actually orchestrating desired scenarios or evaluating test coverage as intended. Finally, none of these techniques offer adequate mechanisms to identify previously unknown hazardous edge case scenarios or aggregate coverage metrics across all virtual and physical testing platforms.

Foretellix, an Israel-based start-up, believes it has a solution. Its Foretify Technology is designed to bring a proven coverage-driven verification approach broadly adopted in the semiconductor industry to the ADAS and AV sectors. Foretify Technology, according to the company, enables the AV industry to move from a focus on Quantity of Miles to a focus on Quality of Coverage.

Foretellix is a member of ASAM (Association for Standardization of Automation and Measuring Systems) – a non-profit organization that promotes standardization for tool chains in automotive development and testing. Members are international car manufacturers, suppliers, tool vendors, engineering service providers and research institutes from the automotive industry. One of the ASAM projects in which Foretellix is involved is the development of the ASAM OpenSCENARIO standard for AVs.

Automotive Industries (AI) asked Ziv Binyamini, CEO and Co-founder, Foretellix, how his company is contributing to the ASAM OpenScenario standard.
Binyamini: Foretellix has developed the Measurable Scenario Description Language (M-SDL) and contributed its concepts to the ASAM OpenScenario 2.0 Standards Concept Project. This project has just recently published its concept document, which uses M-SDL for all examples of a declarative domain specific Scenario Description Language.

Foretellix has also had one of its co-founders act as the lead of the ASAM OpenScenario 2.0 Concept, while its CTO has actively contributed numerous technical contributions. Foretellix brings years of expertise in inventing and defining several verification languages including the “e” verification language (IEEE 1647), UVM and PSS (Portable Stimuli Spec used for verification of Systems on Chips).

AI: How important is standardization for AVs?
Binyamini: As indicated by the very strong interest we’ve had in MSDL, it is clear the industry wants the standard and is ready to adopt it once ratified. Standardization is extremely important for the efficiency of the development process and the support of a vibrant ecosystem that can be readily leveraged by developers. This is why ASAM membership consists of around 300 member organizations worldwide. In addition, over 500 engineers from 250 different entities have already downloaded the M-SDL specification.

The ASAM OpenSCENARIO 2.0 standard is meant to support the definition of tests and scenarios for the full development process of autonomous vehicles, and the full complexity of realworld scenarios, including complex inner-city traffic. Required use cases span from pure software-based simulation, through SIL, HIL, VIL hybrid testing models, up to test tracks and street driving.

AI: Please explain the strategy of the “One to Many” and “Many to One” approach of your Foretify Technology?
Binyamini: Both concepts enable the management of complexity in defining the huge number of scenarios that need to be verified, and the test-related data that must be analyzed. The “One to Many” breakthrough allows “One” scenario specification to represent “Many” meaningful scenarios. The “Many to One” breakthrough allows results from “Many” tests executed across many platforms to be aggregated into “One” coverage dashboard and metric.

This gives the user a quantifiable way to assess where they are in the verification process, what to do next, and when they are done. This enablement is critical because humans cannot deal with the scale of the number of scenarios and amount of test data required to prove an advanced ADAS or Autonomous vehicle is safe.

AI: How well is Foretify meeting its objectives?
Binyamini: Foretify has been refined and proven through multiple proof of concept programs. As we’ve expanded engagements to urban and other specialized applications (e.g., mines), we are continuing to expand capabilities to meet an expanded set of objectives.

AI: How will M-SDL help the developers who build on its framework?
Binyamini: M-SDL is the first open language that addresses multiple shortcomings of today’s formats, languages, methods and metrics used to verify and validate ADAS and autonomous vehicles
(AV), and address the industry mandate for “measurable safety”.

With M-SDL, tool vendors, suppliers and developers are able to:

• Use a common, human readable, high level language to simplify the capture, reuse and sharing of scenarios
• Easily specify any mix of scenarios and operating conditions to identify previously unknown hazardous edge cases; and
• Monitor and measure the coverage of the autonomous functionality critical to prove AV safety, independent of tests and testing platforms.

With M-SDL and Foretify, AV developers gain the combination of productivity, portability and measurability required to address the largest challenges faced by automated driving suppliers and regulators as they pursue the industry mandate of measurable safety.

Productivity is enhanced primarily through the automation of the test generation process, and the ability to reuse high level M-SDL based scenario descriptions across Operational Design Domains (ODDs) and testing platforms (rather than rebuild from scratch).

Portability and the associated benefits of reuse are enabled primarily through the ability to extend and tailor existing M-SDL descriptions to new ODDs, and by leveraging Foretify APIs to integrate multiple simulators and HIL platforms, as well as test track controllers.

Measurability is enabled with intelligent scenario monitoring, data aggregation and the roll up of metrics on a dashboard annotated onto a comprehensive verification plan.

Key metrics include:

AI: How can this improve AV safety?
Binyamini: Today, there are numerous automated driving functions, such as lane keep assist, lane change assist, lane tracing assist, lane centering, adaptive cruise control with stop and go, proximity control, traffic jam assist, autonomous emergency braking and sideguard assist. Unfortunately, many of these driving functions often do not yet perform as intended.

According to a 2018 IIHS report, “Evaluations of adaptive cruise control and active lane-keeping show variable performance in typical driving situations. The outcomes range from the irksome, such as too-cautious braking, to the dangerous, for example, veering toward the shoulder if sensors couldn’t detect lane lines.”

Clearly, suppliers must resolve these issues and improve the verification and validation of these core highway automated driving functions. Foretellix solutions can significantly improve the performance and reliability of the next generation of these systems, which also serve as the foundation for future systems.

As companies build on these core functions, the potential impact on consumer safety grows dramatically.

AI: How does Foretify compare to similar technologies?
Binyamini: Foretify is in a category we call “intelligent verification”. This is contrast with “testing engines” that include simulation and hardware-in-the-loop (HIL). Intelligent verification
offers significant levels of automation and analytics to enable productivity and portability. This enables developers to scale their verification activities to 100s of millions of scenarios, and to realize
portability across test platforms and operational design domains (ODDs). Intelligent verification offerings also enable measurability, with metrics that indicate how thorough testing was done
(coverage), and how well the AV performs in each test.