Research

nxtAIM will leverage the potential of Generative AI for the development of autonomous driving. The project aims to overcome current limitations and provide innovative solutions using new generative methods to master the significant challenges in this field.

Research

nxtAIM will leverage the potential of Generative AI for the development of autonomous driving. The project aims to overcome current limitations and provide innovative solutions using new generative methods to master the significant challenges in this field.

Project goal: What does nxtAIM want to achieve?

The hurdles on the path to higher levels of automated driving remain significant. Particularly concerning scalability (with regard to data and costs), transferability (expansion of Operational Design Domains, ODD), and traceability (safety and acceptance), the current AI methods and tools are reaching their limits. Considering this context, Generative AI offers promising new opportunities and alternative research approaches. Its technological maturity is very high and has impressively demonstrated its capabilities in the last two years with text-to-image generators and large language models.

More Information

nxtAIM seeks to harness the immense potential of Generative AI for the development of autonomous driving. The re-purposing of existing methods and tools could significantly accelerate development. Central to the project are generative, self-supervised learning foundation models as a new paradigm. The partners are leveraging AI for both situational interpretation and planning, as well as for system optimization within a differentiable, bidirectional chain of effects.

Which methods are used?

In the project, Generative AI is used in four research fields of autonomous driving:

1. Generation of multimodal sensor data for training and validation; offline and during operation
2. Generation of traffic scenarios for prediction and planning
3. Development of foundation models and their conditioning
4. System integration and bidirectional information flow: feedback loop for the chain of effects

"Processing of information in the chain of effects of current AD stacks is essentially done without feedback loops. Errors in the perception part can propagate undetected into the path planning. An unrecognized obstacle is interpreted as a clear path. However, the absence of obstacle evidence is not the evidence of absence of an obstacle. For the first time, generative models offer the possibility to establish predictor-corrector methods at the sensor data level, rather than at the level of potentially faulty detections. For example, it would be possible to explicitly check whether a route that is assumed to be free, actually appears free in the camera image."

Dr. Jörg ReichardtnxtAIM project lead, Continental

"Current AI systems require large amounts of vehicle data for training the discriminative methods used, tailored to a specific traffic condition. The costs for creating and validating this data, as well as for labeling, are extremely high. Current approaches therefore lack scalability and transferability, which makes their use expensive and the training of AI models complex. Additionally, the transparency of approaches based on deep neural networks – keyword: black box – remains a significant challenge."

Dr. Ulrich KreßelnxtAIM project co-lead, Mercedes-Benz

nxtAIM research fields and their teams

The work of nxtAIM is organized in six sub-projects. Sub-projects 1, 2 and 3 focus on different parts of the chain of effects. Sub-project 1 aims to generate sensor data from the environment model for individual time steps. Sub-project 2 extends this to include the dynamic development of the situation and aims to create generative models for sequences of sensor data.

Generative models not only enable the sampling of sensor data, but also of time series in general. This is used in subproject 3 to generate traffic scenarios in geometrically abstracted environment models and thus improve prediction and planning algorithms.

Generative models are typically based on a latent feature space. Structuring and interpretation of this space is the task of sub-project 4. Sub-project 5 deals with how the new approaches can be implemented in a system that can be executed in the vehicle. Finally, sub-project 6 will evaluate the systems developed in sub-projects 1, 2 and 3 in terms of their degree of realism.