Knowing About Knowing: An Illusion of Human Competence Can Hinder Appropriate Reliance on AI Systems. (arXiv:2301.11333v1 [cs.HC])

The dazzling promises of AI systems to augment humans in various tasks hinge
on whether humans can appropriately rely on them. Recent research has shown
that appropriate reliance is the key to achieving complementary team
performance in AI-assisted decision making. This paper addresses an
under-explored problem of whether the Dunning-Kruger Effect (DKE) among people
can hinder their appropriate reliance on AI systems. DKE is a metacognitive
bias due to which less-competent individuals overestimate their own skill and
performance. Through an empirical study (N = 249), we explored the impact of
DKE on human reliance on an AI system, and whether such effects can be
mitigated using a tutorial intervention that reveals the fallibility of AI
advice, and exploiting logic units-based explanations to improve user
understanding of AI advice. We found that participants who overestimate their
performance tend to exhibit under-reliance on AI systems, which hinders optimal
team performance. Logic units-based explanations did not help users in either
improving the calibration of their competence or facilitating appropriate
reliance. While the tutorial intervention was highly effective in helping users
calibrate their self-assessment and facilitating appropriate reliance among
participants with overestimated self-assessment, we found that it can
potentially hurt the appropriate reliance of participants with underestimated
self-assessment. Our work has broad implications on the design of methods to
tackle user cognitive biases while facilitating appropriate reliance on AI
systems. Our findings advance the current understanding of the role of
self-assessment in shaping trust and reliance in human-AI decision making. This
lays out promising future directions for relevant HCI research in this
community.