Using AI Uncertainty Quantification to Improve Human Decision-Making. (arXiv:2309.10852v1 [cs.AI])

AI Uncertainty Quantification (UQ) has the potential to improve human
decision-making beyond AI predictions alone by providing additional useful
probabilistic information to users. The majority of past research on AI and
human decision-making has concentrated on model explainability and
interpretability. We implemented instance-based UQ for three real datasets. To
achieve this, we trained different AI models for classification for each
dataset, and used random samples generated around the neighborhood of the given
instance to create confidence intervals for UQ. The computed UQ was calibrated
using a strictly proper scoring rule as a form of quality assurance for UQ. We
then conducted two preregistered online behavioral experiments that compared
objective human decision-making performance under different AI information
conditions, including UQ. In Experiment 1, we compared decision-making for no
AI (control), AI prediction alone, and AI prediction with a visualization of
UQ. We found UQ significantly improved decision-making beyond the other two
conditions. In Experiment 2, we focused on comparing different representations
of UQ information: Point vs. distribution of uncertainty and visualization type
(needle vs. dotplot). We did not find meaningful differences in decision-making
performance among these different representations of UQ. Overall, our results
indicate that human decision-making can be improved by providing UQ information
along with AI predictions, and that this benefit generalizes across a variety
of representations of UQ.



Related post