In 2018, as self-driving cars started rolling cautiously through the streets of Silicon Valley and select cities, Stanford’s own experts said we were 90 percent of the way to a driverless future. But that last 10 percent proved stubborn. While tech optimists were promising self-sufficient planes, trucks, and cars, researchers were still trying to figure out how these vehicles could see through fog and rain, navigate using street markings of varying styles and quality, and learn safe decision-making.

Cut to today: Cars without drivers – or even without a driver’s seat – are now operating routinely in several cities, including in the San Francisco Bay Area.

“After approximately 20 years of development, AV technology is really becoming part of our daily lives,” said Marco Pavone, associate professor of aeronautics and astronautics in the School of Engineering.

Marco Pavone | Rod Searcey

Pavone, who is director of the Autonomous Systems Laboratory and the Sustainable Mobility Center, has been in the thick of autonomous robotic research and development. He shared insights about what has gotten this technology across the finish line and what to expect next.

Where are we with autonomous vehicle technology?

Here in the Bay Area and San Francisco, robotaxi services are becoming quite mainstream, not just in the cities but also in the suburbs. They aren’t providing the majority of ridership across the United States yet, but if you go to San Francisco, chances are you will see one. Advanced automated technologies, such as autopilot, are becoming mainstream on consumer vehicles as well.

This is one of the most complex technologies that humankind has developed, so it took quite some time to perfect it. But there has been an acceleration lately, and AI has been key.

AI has accelerated progress across the development cycle. It advanced the software on the vehicle and also improved all the processes that are necessary to build that software, such as curating data for model training, improving the fidelity of simulation, and automating evaluations of the vehicle’s performance.

What’s one way AI has made driverless cars more road-ready?

To enable autonomy, cars have software stacks that address and coordinate – either explicitly or implicitly – various tasks, like object detection, object tracking, planning, control, and decision-making. In the past few years, there’s been a shift from engineers manually coding individual pieces of software and integrating them into a cohesive stack toward increasingly data-driven and end-to-end systems. Neural networks now play a much larger role across the stack, with models increasingly learning from large-scale driving data, including demonstrations collected from human drivers.

This change has been proven very useful in a number of ways. First of all, the ceiling for data-driven approaches is much higher: With more high-quality and diverse data, these systems can continue to improve. Also, some tasks are very hard for engineers to recreate from scratch. Whereas, if the vehicle learns from human demonstrations, the driving is more likely to align with how humans actually drive.

What are some of the toughest problems to solve when it comes to AV technology?

Two key bottlenecks are deploying autonomous vehicle systems more widely and validating the safety of these systems – without requiring excessive amounts of new data.

Robot autonomy means developing robotic systems that can work in situations that cannot be foreseen at the design stage; systems that can reason about complex situations in some intelligent way. As we try to design for more and rarer situations, it becomes increasingly expensive and harder for human designers to conceptualize those. So, you’re basically relying on driving so many miles that virtually every situation can be experienced and taught to the system.

As for safety, you don’t just want a system that is safe and robust, you also have to prove to yourself and to the community that it is safe and dependable. To do that, you need technologies that can robustly validate the system with an attainable amount of data.

One of the new, exciting developments to address both challenges has been leveraging internet-pretrained models. I compare this to a human learning how to drive: When you’re practicing for your driver’s license, you rely on 16 years of experience gained through tasks not related to driving – being a passenger, being a pedestrian, improving your reaction time through sports, etc. – that scaffold the core capabilities needed to become a good driver. That’s why, depending on the state, you can qualify for a driver’s license with only tens of hours of supervised driving practice. Similarly, internet-pretrained models can leverage knowledge acquired from a vast range of tasks that are not directly related to driving.

This provides the vehicle with a broad base of prior knowledge and reasoning capabilities, so that driving-specific data can be used to specialize the model rather than teach it everything from scratch. In other words, by leveraging experience from the broader internet, we can substantially reduce the amount of driving data needed to make the vehicle a capable driver. This can also help with safety validation – for example, by generating challenging yet plausible scenarios that can be used to stress-test the vehicle and uncover failure modes that may be difficult to discover from driving data alone.

What’s next for autonomous vehicle technology?

Automated technologies are a spectrum. In the next two or three years, we will see advanced automated technologies deployed across nearly every type of vehicle beyond passenger cars: trucks, construction machines, agricultural machines, low-speed shuttles, and the various vehicles that move people in airports. So, even though we are witnessing automated technologies most visibly in the context of personal mobility, it is going to be widespread – and this is happening as we speak. These technologies will make vehicles safer and more comfortable.

The big next question is: Where else are we going to see robotic systems? Humanoids? Quadrupeds? The timeline for such things will be much more varied, given the significant fragmentation of the robotics ecosystem across tasks, embodiments, and industries. But it’s plausible that, sooner rather than later, this new wave of robotics will find viable applications and begin to reach significant economic scale.

The road to autonomy

With safety as a guiding principle, Stanford research has long advanced the science of autonomous driving. Here’s a look back at some of the ideas and experiments behind that progress.

Testing the limits with a drifting DeLorean (2015)
Engineers built MARTY, a self-driving research vehicle, to study the ways cars handle extreme situations.

Understanding the handoff to human drivers (2016)
Researchers tested how drivers adjusted their steering when retaking control under changed conditions.

Giving robots a wider view (2017)
Stanford and UC San Diego researchers developed a wide-angle 4D camera with potential applications in autonomous vehicles and other robotic systems.

Connecting space robotics and self-driving cars (2017)
Marco Pavone explored how algorithms for robots in space could help autonomous vehicles navigate changing environments on Earth.

Seeing around corners (2018)
Researchers developed a laser-based imaging technique to reveal hidden objects, with potential applications for detecting road hazards.

Coordinating two driverless cars in a tandem drift (2024)
Stanford and Toyota Research Institute researchers demonstrated autonomous tandem drifting to explore ways AI could improve driving safety.