Machine learning models can make mistakes and be difficult to use, so scientists have developed explanation methods to help users understand when and how to trust a model’s predictions.
However, these descriptions are often complex and contain information about perhaps hundreds of model features. Additionally, they may be presented as multifaceted visualizations, which can be difficult for users without machine learning expertise to fully understand.
To make AI explanations understandable, MIT researchers used large-scale language models (LLMs) to translate plot-based explanations into plain language.
They create a two-part system that converts machine learning descriptions into paragraphs of human-readable text and then automatically evaluates the quality of the description so that end users can decide whether they can trust it. Developed.
By presenting the system with several example explanations, researchers can customize the explanations to suit user preferences and specific application requirements.
In the long term, the researchers hope to further develop this technology by allowing users to ask the model follow-up questions about how it came up with its predictions in real-world settings. Masu.
“Our goal with this research is to take the first step toward allowing users to have full-fledged conversations with machine learning models about why they made certain predictions, and whether they should listen to the model at all.” “It was about being able to make better decisions,” he says. Alexandra Zytek is a graduate student in electrical engineering and computer science (EECS) and lead author of a paper on this technology.
She is joined on the paper by MIT postdoc Sara Pido. Sarah Arnegeimisch, EECS graduate student. Laure Berti-Equille, research director at the French National Institute for Sustainable Development. Senior author Kalyan Veeramachaneni is a Principal Research Scientist at the Institute for Information and Decision Systems. The research will be presented at the IEEE Big Data Conference.
Easy to understand explanation
The researchers focused on a common type of machine learning explanation called SHAP. A SHAP description assigns a value to every feature that the model uses to make predictions. For example, if your model predicts house prices, one of the features might be the location of the house. A position is assigned a positive or negative value that represents how much that feature changes the overall model prediction.
SHAP descriptions are often displayed as a bar chart showing which features are most or least important. However, for models with more than 100 features, the bar graph quickly becomes unwieldy.
“As researchers, we have to make many choices about what to present visually. If we only show the top 10, we wonder what happened to the other features that are not in the plot. “Using natural language reduces the burden of making those choices,” says Veeramachaneni.
However, rather than leveraging large-scale language models to generate descriptions in natural language, researchers use LLM to transform existing SHAP descriptions into easy-to-read narratives.
By having LLM handle only the natural language portion of the process, Zytek explains, they limit the potential for inaccuracies in the description.
Their system, called EXPLINGO, is divided into two parts that work together.
The first component, called NARRATOR, uses LLM to create narrative descriptions of SHAP explanations tailored to the user’s preferences. You can first give NARRATOR three to five examples of story descriptions, and LLM will mimic those styles when generating text.
“Rather than trying to define the type of description the user wants, it’s easier to just have them write what they want to see,” Zytek says.
This allows you to easily customize NARRATOR for new use cases by presenting a different set of manually created samples in NARRATOR.
After NARRATOR creates a plain description, the second component, GRADER, uses LLM to rate the story on four metrics: brevity, accuracy, completeness, and fluency. GRADER automatically displays text from NARRATOR and the SHAP description it describes in LLM.
“We found that even if LLMs made mistakes in performing a task, they often did not make mistakes when checking or validating that task,” she says.
Users can also customize GRADER to give different weights to each metric.
“For example, in a high-stakes case, you would think that accuracy and completeness would be much higher than fluency,” she added.
Story analysis
For Zytek and his colleagues, one of the biggest challenges was tuning LLM to generate natural-sounding stories. The more guidelines you add to your control style, the more likely LLM will introduce errors into your description.
“Many quick adjustments were made to find and correct mistakes one by one,” she says.
To test the system, the researchers obtained nine machine learning datasets with explanations and had different users write explanations for each dataset. This allowed us to assess the narrator’s ability to imitate a unique style. They used GRADER to score each narrative description on all four indicators.
Ultimately, the researchers found that their system produced high-quality narrative descriptions and could effectively mimic a variety of writing styles.
Their results show that providing several manually written example explanations can significantly improve narrative style. However, these examples should be written carefully. Including comparative words such as “larger” can cause GRADER to mark accurate descriptions as inaccurate.
Based on these results, the researchers hope to explore techniques that allow the system to better handle comparative terms. We also want to extend EXPLINGO by streamlining the explanations.
In the long term, we hope to use this research as a stepping stone to an interactive system where users can ask the model follow-up questions about its explanations.
“That will help decision-making in many ways. When people disagree with the model’s predictions, we can figure out whether their intuition is correct, whether the model’s intuition is correct, and where the difference is coming from. We want it to be easy to understand,” says Zytek.
