nxtAIM in dialogue with Dr. Robert Habeck, Federal Minister for Economic Affairs and Climate Action
Autonomous Driving and AI: A nxtAIM Presentation
The nxtAIM project, presented by coordinator Dr. Jörg Reichardt at the Ministry for Economic Affairs and Climate Protection on October 7, 2024.
The presentation is also available as a video stream at:
The Mobility of the Future
The future of mobility and logistics is autonomous. Trucks will transport goods from A to B 24 hours a day, seven days a week, instead of their drivers moving from rest area to rest area. The control of a passenger car will be taken over by the vehicle itself, turning the driver into a passenger.
Function Development
The ability to navigate autonomously relies largely on machine learning methods and artificial intelligence. Functionality is learned by a computer system based on precisely annotated example or training data. Collecting the necessary training data in real-world traffic and creating their annotations—so-called labels—is a costly and complex task. This is especially true for rare and difficult but highly relevant scenarios in which even human drivers face significant challenges.
What can generative AI contribute to the development of autonomous driving functions? Generative models are also trained on a sample of example data. However, they do not learn a specific function necessary for autonomous driving, such as pedestrian or lane recognition, from labeled examples.
Generative AI
Generative models are stochastic models. They learn to describe the distribution of example data in a way that enables sampling from it. Using random stochastic processes, they generate new, synthetic data equivalent to the original training data.
In function development, we are no longer limited by the amount of collected real-world data but can instead generate an unlimited amount of new data through sampling. This opens entirely new possibilities for training, testing, and validating driving functions, particularly in the difficult edges of the operating range – the “edge cases”. Generative models thus enable the gradual expansion of functional capabilities. When used in future vehicles, they enable the real-time validation of perception by comparing environmental hypotheses with the actual sensor data.
Success Factors
The nxtAIM project brings together three key success factors. First, the technical maturity of generative methods. Examples of improbable but photorealistic images, such as the Pope in a white puffer jacket, illustrate how generative AI can create highly realistic images. If humans can no longer distinguish a generated image from a real photo, then the quality of generated data is high enough to train and test perception systems. If this works with camera data, it should also be possible with other sensor modalities such as radar or LiDAR. Since generative methods do not require labels or annotations and are trained independently of specific functions, they can be scaled relatively easily to vast datasets, given sufficient computing power.
Furthermore, generative methods are remarkably versatile. They can also be used to model traffic scenarios and predict the behavior of road users—an essential requirement for path planning.
The second success factor is the ability to control and guide the generation process according to specific requirements. In certain aspects—such as the posture of people in a scene—precise specifications can be made, while other elements like clothing and background remain randomly generated.
This controllability can also be added retrospectively to already trained models, requiring significantly less data and computational effort than training the original model. This is a defining characteristic of basic or foundation models.
This also marks a paradigm shift in AI development: from models designed for specific functions based on function-specific data and annotations to foundation models with broad applicability derived from diverse data sources.
The third success factor is the wealth of data collected by industry partners over recent years. These datasets are now being utilized in this project to develop generative models tailored to automotive sensor requirements, all while adhering to strict data protection regulations.
Finally, the consortium’s expertise—combining academic and industrial knowledge within the VDA flagship initiative—plays a crucial role. The cooperation with the Jülich Supercomputing Center as a project partner, which provides the computer infrastructure required for the project, should be highlighted here. The active exchange with scientific partners, combined with the required computing power, will have a long-term positive impact on the industry’s competitiveness.
Notably, some of the foundational technologies for this project were already developed within the initiative. Stable Diffusion, one of the most well-known text-to-image generators, was developed at Heidelberg University and LMU with funding from the LI project KI-Absicherung.
The project partners expect that the generative models developed within nxtAIM will further advance the successful development of automated driving functions and complement existing architectures. As stochastic models, they seamlessly enhance classical simulations for synthetic data generation, introducing an unprecedented level of variation. The nxtAIM team is convinced that this project will significantly accelerate progress toward autonomous driving.
Verifying Safety
However, for the vision of autonomous mobility of the future to become a reality, technical excellence alone is not enough. If the person behind the wheel is to truly relinquish responsibility, that responsibility must be assumed elsewhere.
A person who lets go of the steering wheel but still bears responsibility for the driving function is not sitting in a self-driving car. Instead, they are merely being forced to permanently monitor a technical system.
The German automotive industry takes a different approach: it assumes responsibility and liability. The first internationally approved Level 3 systems for private vehicles—where the driver transfers both control and legal responsibility to the manufacturer, truly becoming a passenger—come from Germany.
The project team firmly believes that only this consistent approach can build trust. Only trust leads to acceptance, and ultimately, only acceptance will create dissemination. This is the path that nxtAIM will continue to pursue.
