The newly-launched Police Records Access Project makes information about California police misconduct and use of force publicly searchable for the first time in the state’s history – a significant move toward improving transparency in law enforcement.

The database, funded by the State of California, contains 1.5 million pages of once-secret police records on approximately 12,000 cases involving nearly 500 law enforcement and oversight agencies across the state, expanding visibility into allegations of misconduct and uses of force resulting in death or serious injury. Stanford’s Big Local News worked with UC Berkeley Journalisms Investigative Reporting Program and the Berkeley Institute for Data Science to build the database, which was published by CalMatters, KQED, the Los Angeles Times, and the San Francisco Chronicle.

“Police records especially can often be opaque to the general public,” said Cheryl Phillips, founder of Big Local News and a Hearst Professional in Residence at Stanford’s School of Humanities and Sciences. “This database of police records is critical for an agency’s credibility and transparency. It is the first of its kind in the nation and allows light to be shed on systemic issues.”

Big Local News, part of the School of Humanities and Sciences’ Department of Communication, compiles difficult-to-access government data and works with local newsrooms on investigative reporting projects. The Police Records Access Project reflects years of work and a unique collaboration of more than a hundred journalists, data scientists, lawyers, and civil liberty advocates across the state.

Related story

Stanford collaboration contributes to Pulitzer Prize-winning reporting

Stanford’s Big Local News and Libraries supported a Pulitzer-winning investigation into Baltimore’s fentanyl crisis, helping journalists uncover overdose patterns among Black men.

Pooling resources

The 2018 passage of California Senate Bill 1421, also known as “The Right to Know Act,” sparked the beginning of the project by making police records on misconduct and use of force available to the public. In anticipation of the massive amount of data that would soon become accessible, journalists from six newsrooms began working together to request and share the records.

Senate Bill 16, approved in 2021, further expanded access to police officers’ personnel records. Both laws were introduced by then-state Senator Nancy Skinner, who later helped secure state funding to create the database.

The project soon grew to include 40 newsrooms, which formed the California Reporting Project. The partnership allowed participating media outlets to share leads and pool scarce resources, which was particularly helpful for newsrooms that were also juggling major stories such as California’s massive wildfires, the COVID-19 pandemic, the 2020 election, and civil unrest.

“Local journalism organizations are strapped for resources, so collaboration is how we have to operate now,” Phillips said. “It was a pretty unique and first-of-its-kind collaboration on such a broad level. It involved a lot of innovative problem solving.”

While the records were technically accessible by the public after the passage of the bills, it remained onerous to request records from one agency at a time, and impossible to identify statewide trends and patterns.

Some law enforcement agencies also refused to provide records or began destroying them, prompting lawsuits. To help address these challenges, the project included the help of the ACLU Foundation of Southern California, California Innocence Network organizations, the National Association of Criminal Defense Lawyers, UC Irvine Law School’s Press Freedom Project, and UC Berkeley Law Schools Criminal Law & Justice Center.

In 2020, the project team built the first edition of the database, which included a searchable user interface. Stanford and Berkeley also helped journalists file records requests, identify story possibilities, manage records, and offered reporters trauma training as they read through hundreds of pages of often disturbing incidents.

“We did anything to facilitate getting stories out to the public,” Phillips said. “I don’t think this project could have happened without university involvement, from the data scientists to PhD students developing new statistical measures for discrimination.”

Phillips said it was also important to ensure that journalists working on stories remained independent from and uninfluenced by the organizations helping to build the project’s infrastructure.

“Getting the logistics of that right was important and hard,” Phillips said. “But the other thing is that the records themselves are daunting with thousands of cases.”

Puzzle pieces

Since the launch of the database, more than 3,500 requests have been made to law enforcement agencies, including police and corrections departments, sheriff’s offices, and district attorneys’ offices. (Federal agencies are not subject to the state transparency laws and are not included in the database.) Many of the requested records pertain to cases between 2010 and 2020, in part because information requests cannot be made on open investigations.

One of the project’s biggest challenges has been piecing records together as agencies provide them, said Lisa Pickoff-White, the director of research at the Berkeley Investigative Reporting Program’s California Reporting Project. Pickoff-White joined the project in 2018 as a data journalist at KQED, and later spent a year as a fellow at Stanford’s Big Local News, where she focused on using rapidly improving AI tools to help process more than 22 terabytes of data. For example, the project recently received a hard drive with files related to a request made two years ago.

“It was like we were doing 1,000 puzzles at once,” Pickoff-White said. “Someone would send us puzzle pieces, sometimes literally in the mail, and then we have to figure out which puzzle the pieces go to before we can put them together.”

The project team used AI tools to extract basic information from text files or images. The database doesn’t include audio recordings or videos at this time. The team also removed or redacted graphic imagery and personal information about victims of sexual assault and domestic violence from the files.

The improved transparency of police records has made it possible for journalists like Pickoff-White to identify trends and produce stories, such as one she worked on with Stanford and Berkeley journalism students that focused on uses of force that resulted in broken bones. More than 100 stories have been produced to date as a result of the project.

“We really have these laws because of victims of police violence and their families,” Pickoff-White said. “Previously, the law did not allow them to access records about their loved ones or themselves, and they wanted to know what happened. This database also allows researchers to avoid going to every single jurisdiction to gather up police reports and research different trends so they can make policy recommendations.”

Researchers have expressed interest in using the database to explore topics such as encounters between police and people who are mentally ill, she said. The project team also hopes that law enforcement agencies will use the information when hiring between agencies.

California Reporting Project journalists gather at a National Institute for Computer-Assisted Reporting (NICAR) conference in Nashville on March 2, 2023. | Courtesy Lisa Pickoff-White

Additional newsrooms are expected to publish the database as part of a broader rollout, and the project team plans to add new features and tools as they are developed. They are also speaking with counterparts in other states about how to replicate the database.

“It’s bringing the right combination of people together from different industries to create useful tools together,” Pickoff-White said.