![Cover image for [AWS] The Future of Automobiles and Automated Driving [Generative AI]](https://media2.dev.to/dynamic/image/width=1000,height=420,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fgbh4a3u5rcanbnc4c1wy.jpg)
This post introduces the contents of one session at AWS Summit Japan 2024. The content is about how AWS and the latest generated AI technologies relate to the automotive industry.
Conventional Automotives
Currently, the development of information and AI technology is bringing about a significant shift in the way cars are made. During this period of transformation, the concept of the Software-Defined Vehicle (commonly known as SDV) has appeared, and car manufacturers are dramatically changing the way they build cars in line with this concept. Furthermore, the recent boom in generative AI is opening new paths for autonomous driving technology. This includes enhancing autonomous driving technology using LLMs (Large Language Models).
Development of automated driving technology using LLMs
Conventional automated driving technology
The below slide explains how the methods of achieving autonomous driving are changing.
At the AWS Summit, Turing Corporation introduced that the way of achieving autonomous driving is fundamentally changing. Until now, the mainstream method combines LiDAR and radar technologies to acquire and interpret surrounding information. However, currently, companies like Tesla and Turing Corporation are leading the research and development of systems that make decisions based solely on images captured by cameras, and they are fiercely competing in this field.
Thanks to the advancements in deep learning technology, autonomous driving is becoming possible using only cameras, without the need for LiDAR or radar. Compared to LiDAR and other sensors, cameras are relatively inexpensive, and their mechanism for situational judgment is closer to that of human drivers. Ideally, a camera-based system would be preferable.
Contribution of Natural Language Models to Automated Driving
According to Li's research1, autonomous driving technology is evolving towards a more advanced understanding of situations by using natural language models. This is largely due to the recent remarkable advancements in LLMs.
The above slide explains that the learning tasks for autonomous driving are shifting towards those that use situational understanding through natural language.
However, LLMs cannot be directly used for autonomous driving because they cannot interpret images as they are.
That's where multimodal models come in
Multimodal models are learning models capable of processing various types of data sources. Turing Corporation has introduced a method of further training advanced logical thinking LLMs with driving data such as videos. This approach enables the creation of multimodal models specialized for autonomous driving.
The above slide explains the way multimodal models can be trained quickly in large-scale distributed environments.
AWS resources used for training
Turing Corporation achieved large-scale training of multimodal models by clustering AWS-provided P5 instances. In addition, they used AWS ParallelCluster for clustering, they were able to create an environment similar to a supercomputer.
It might seem like an expensive setup, but in 2023, they participated in the AWS LLM Development Support Program hosted by AWS Japan, a program where AWS covers part of the costs incurred during the development of LLMs. This allowed them to conduct the training.
The completed multimodal model
The model created using the above method is available as open source! It is called Heron.
The above slide explains that by converting images into language tokens, LLMs are now able to interpret images.
Methods of collecting training data
Data collection is also carried out using AWS resources. Various sensor data is collected using AWS IoT Greengrass. Data cleaning and sampling are also completed on AWS.
Designing the architecture for running LLMs
Furthermore, Turing Corporation is also designing the architecture for running the developed multimodal model. This development utilizes FPGAs.
What's an FPGA?
An FPGA (Field Programmable Gate Array) is a circuit that does not have a predefined instruction set and can be reconfigured using a program. An FPGA contains numerous internal logic blocks and variable interconnects that can be configured through programming. This allows for the flexible construction of circuits with the necessary instruction sets, enabling the creation of circuits capable of performing operations such as the Softmax function used in LLMs.
EC2 F1 instance
AWS offers EC2 instances equipped with FPGAs. This allowed Turing Corporation to conduct preliminary verification of their architecture design on AWS before developing actual FPGAs in a production environment.
A bonus story about SDVs
From here, I'll share a bit about SDVs that were introduced in a separate session.
Traditionally, major functions and services in automobiles have been controlled by dedicated hardware. Engine, transmission, braking system, steering, and other functions operated independently, often requiring physical parts replacement or repairs for modifications. The characteristics of hardware-based development are as follows:
- Adding or improving functions requires the introduction of new hardware or physical modifications.
- Update frequency is low, and incorporating new technology takes time and costs.
- The role of software is limited, primarily used to control hardware.
On the other hand, SDVs define and control the vehicle's major functions and services through software. This new approach offers flexibility and quick updates. The characteristics of SDVs are as follows:
- New functions and services can be added or improved through software updates, eliminating the need for physical parts replacement.
- Real-time updates are possible via over-the-air (OTA) updates.
- Data-driven development ensures that the vehicle is always up to date, enhancing the user experience.
A story that personally excited me
In the future, autonomous driving technology may develop further, and we might use autonomous driving in our daily lives. However, I personally enjoy driving myself, so I might feel a bit sad if I could only drive an autonomous car.
But if SDV becomes mainstream, we might be able to realize use cases like the following:
- Autonomous driving for everyday use
- Switching the car to a racing tune for enjoying motorsports on weekends!
Imagine if car manufacturers provide a page on AWS where users can configure various settings for the motor and electronic control suspension, allowing the vehicle's behavior to change dynamically. Wouldn't that be incredibly exciting?
As a fan of Gran Turismo, the thought of being able to freely tune my car excites me greatly!
Conclusion
The automotive environment is rapidly changing and evolving, driven by the advancements in cloud and AI technology. What is considered cutting-edge technology today might become outdated a year from now. This rapid pace of evolution makes us feel that way.
Moreover, the concept of SDVs is expected to offer us users a more flexible and enjoyable way to experience cars. Behind the scenes, cloud providers like AWS are likely playing a crucial supportive role.
Top comments (0)