From the understated opulence of a Bentley to the stalwart family minivan to the utilitarian pickup, Americans know that the car you drive is an outward statement of personality. You are what you drive, as the saying goes, and researchers at Stanford have just taken that maxim to a new level.

Using computer algorithms that can see and learn, they have analyzed millions of publicly available images on Google Street View. The researchers say they can use that knowledge to determine the political leanings of a given neighborhood just by looking at the cars on the streets.

Fei-Fei Li, associate professor of computer science (Image credit: L.A. Cicero)

“Using easily obtainable visual data, we can learn so much about our communities, on par with some information that takes billions of dollars to obtain via census surveys. More importantly, this research opens up more possibilities of virtually continuous study of our society using sometimes cheaply available visual data,” said Fei-Fei Li, an associate professor of computer science at Stanford and director of the Stanford Artificial Intelligence Lab and the Stanford Vision Lab, where the work was done.

Li is an expert in computer vision and deep learning, a type of artificial intelligence in which computers teach themselves to recognize three-dimensional objects in two-dimensional images – computers that see, as she describes them.

The algorithms were trained – or more accurately, they trained themselves – to recognize the make, model and year of every car produced since 1990 in each of more than 50 million Google Street View images from 200 American cities.

The data on car types and location were then compared against the most comprehensive demographic database in use today, the American Community Survey, and against presidential election voting data to estimate demographic factors such as race, education, income and voter preferences.

Li and her team found a simple linear relationship exists between cars, demographics and political persuasion. The societal associations were “simple and powerful,” as the authors describe in their paper published Nov. 28 in the Proceedings of the National Academy of Sciences.

For instance, if the number of sedans in a neighborhood is greater than the number of pickups, there is an 88 percent chance that the precinct will vote Democratic. Transpose those numbers to have more pickups than sedans and there is an 82 percent chance a precinct will vote Republican.

Timely and continuous

Beyond the obvious political repercussions, the researchers believe their algorithms could help provide more timely and continuous supplements to current demographic surveys. The American Community Survey is now conducted through expensive and labor-intensive door-to-door canvassing costing the U.S. more than $250 million annually. Even at that cost, what is worse is the lag time between data collection and publication, which  stretches two years or more, especially for small cities and rural areas.

By comparison, Li’s work piggybacks on a publicly available, regularly updated image database, built and paid for by Google Street View, and it generates analyses in near real time.

“I don’t see something like this replacing the American Community Survey, but as a supplement to keep the data up to date,” said Timnit Gebru, first author of the paper and formerly a member of Li’s lab. Gebru is now a postdoctoral researcher in the Fairness Accountability Transparency and Ethics (FATE) in Artificial Intelligence group at Microsoft Research.

Getting to this point was not easy, Gebru said. The team first had to build by hand an image database of all cars since 1990 – year, make, model, trim packages – and then teach a computer to recognize the subtle differences between cars in partially obscured and odd-angle images.

Subtle differences

They began with a database of 15,000 cars from car-sale website Edmunds.com, but that was only the start. Human experts next had to categorize the cars to the subtlest detail, one-by-one. The difference between the 2007 and 2008 Honda Accord, for instance, is an almost-imperceptible change to the rear taillights.

The algorithm worked fast, taking just two weeks to sort the cars in all 50 million images into 2,657 categories by make, model and year. A human working at a relatively high rate of six images a minute would need 15 years to complete the same task.

Some outside the research group were skeptical, Gebru said. They pointed to inconsistencies in the times of day when images were taken that might affect traffic patterns and vehicle makeup. In fact, Gebru adds, many Street View images are taken in the early morning hours specifically to avoid traffic, providing some consistency to the time of day when the images were taken. Regardless of the level of traffic, Gebru said, the images provided valuable data.

Google Street View image of a residential neighborhood in San Francisco. (Image credit: Google Street View)

“If you walk around a neighborhood looking at cars, the density of traffic sometimes tells you things as valuable as the types of cars you see on the streets,” Gebru said. “We can use all this information in our algorithms.”

Gebru has high hopes for her new application. She is excited to move beyond demographics and use visual imagery analysis to improve surveys of hard-to-reach areas or for other beneficial uses like monitoring carbon dioxide levels and easing traffic congestion.

Li concurred. “It can help us understand how our society works, the things people need and how we can improve lives,” Li said. “There is great potential to use computer vision technology in a constructive and benevolent way.”

Li is also a member of Stanford Bio-X and the Stanford Neurosciences Institute. Other Stanford authors include graduate students Jonathan Krause, Yilun Wang, Duyun Chen, and colleagues from the University of Michigan and Baylor University.

The research was supported in part by funding from the National Science Foundation and the Stanford DARE fellowship and with GPUs donated by NVIDIA.