A new study from researchers at MIT and Pennsylvania State University finds that large-scale language models used in home surveillance could recommend calling the police even when no criminal activity is captured on surveillance video.
Moreover, the models the researchers studied were inconsistent about which videos to flag for police intervention. For example, a model might flag one video showing a vehicle break-in but not another video showing the same activity. The models often disagreed about whether the police should be called in response to the same video.
Additionally, the researchers found that after accounting for other factors, some models flagged videos requiring police intervention less frequently in predominantly white neighborhoods, suggesting that the models exhibit inherent bias influenced by neighborhood demographics, the researchers said.
These results suggest that the model is inconsistent in how it applies social norms to surveillance videos that depict similar activity. This phenomenon, which the researchers call norm inconsistency, makes it difficult to predict how the model will behave in different situations.
“We need to think more carefully about this move fast and break things approach of deploying generative AI models everywhere, especially in high-risk situations, because it could be very harmful,” said co-senior author Assia Wilson, the Lister Brothers Career Development Professor in the Department of Electrical Engineering and Computer Science and principal investigator in the Institute for Information and Decision Systems (LIDS).
Moreover, researchers don’t have access to the training data or inner workings of these proprietary AI models, so they can’t pinpoint the root causes of norm inconsistencies.
Large-scale language models (LLMs) may not currently be deployed in real-world surveillance situations, but they are used to make normative decisions in other important contexts, such as healthcare, mortgage lending, and employment, and in these contexts the models are likely to exhibit similar inconsistencies, Wilson says.
“There’s an implicit belief that law students have already learned or will be able to learn certain norms and values. Our study shows that this isn’t the case — all they’re learning may be arbitrary patterns or noise,” says Shomik Jain, a graduate student at the Institute for Data, Systems, and Society (IDSS) and lead author on the paper.
Wilson and Jain were joined on the paper by co-lead author Dana Caracci, PhD (Class of 2023), an assistant professor in Penn State’s School of Information Science and Technology. The research will be presented at the AAAI Conference on AI, Ethics and Society.
“A real, immediate and actual threat”
The research grew out of a dataset containing thousands of home surveillance videos from Amazon Ring that Caracci built in 2020 while he was a graduate student at the MIT Media Lab. Ring, a maker of smart home surveillance cameras that was acquired by Amazon in 2018, offers customers access to a social network called “Neighbors” where they can share and discuss videos.
Caracci’s previous research has shown that people sometimes use the platform to “racially police territory,” judging who does and doesn’t belong there based on the skin color of the video’s subjects. She planned to train an algorithm to automatically caption videos to find out how people use the Neighbors platform, but the existing algorithm at the time was not good enough at captioning.
The project has changed direction due to the proliferation of LLMs.
“There’s a real, immediate, practical threat that someone is using an off-the-shelf generative AI model to review the video, warn the homeowner, and automatically call the police. We wanted to understand how dangerous that could be,” Calacci says.
The researchers chose three LLMs – GPT-4, Gemini, and Claude – and showed the models real videos posted to the Neighbors platform from the Calacci dataset. They asked the models two questions: “Is a crime occurring in the video?” and “Does the model recommend calling the police?”
The researchers had humans annotate the videos to identify day or night, type of activity, and the gender and skin color of the subjects, and they also used census data to gather demographic information about the areas where the videos were taken.
Inconsistent decisions
They found that even though 39 percent of the models showed a crime, all three models almost always said no crime was occurring in the videos or gave ambiguous responses.
“Our hypothesis is that the companies that develop these models are taking a conservative approach by limiting what their models show,” Jain says.
But while the model found that most videos did not contain any crime, it still recommended calling the police on 20-45% of videos.
When the researchers looked more closely at neighborhood demographics, they found that, controlling for other factors, majority-white neighborhoods were less likely to recommend calling the police in some models.
They found this surprising because the models had not been given any information about the neighborhood’s demographics, and the video only showed the area a few yards from the home’s front door.
In addition to asking the models about the crimes in the videos, the researchers also asked them why they made the choices they did. Examining the data, they found that models in predominantly white neighborhoods were more likely to use terms like “delivery man,” but in neighborhoods with a higher percentage of residents of color, they were more likely to use terms like “burglary equipment” and “property inspection.”
“Maybe there’s something in the context of these videos that gives the model an implicit bias. There isn’t a lot of transparency about these models or the data they’re trained on, so it’s hard to determine where these inconsistencies are coming from,” Jain says.
The researchers were also surprised that the skin color of people in the video didn’t have a significant effect on whether the model recommended calling the police, which they speculate is because the machine learning research community has focused on mitigating skin color bias.
“But when you find countless biases, they’re hard to control. It’s like a game of whack-a-mole: you reduce one, but another one pops up somewhere else,” Jain says.
Many mitigation techniques require knowledge of bias first, and while these models, when deployed, might allow companies to screen for skin color bias, bias based on neighborhood demographics would likely go completely unnoticed, Caracci adds.
“There’s a proprietary assumption that models can be biased, and companies test them before deploying them. Our results show that this is not enough,” she says.
To that end, one of the projects Caracci and her collaborators want to work on is a system that lets people more easily identify bias and potential harm in AI and report it to companies or government agencies.
The researchers also want to study how the normative judgments LLMs make in critical situations compare to human judgments, and what facts LLMs understand about these scenarios.
This research was funded in part by IDSS’s Ending Systemic Racism Initiative.
