Re-evaluating Theory of Mind Evaluation in Large Language Models

Theory of Mind (ToM) is the human ability to reason about others' mental states, crucial for social interaction, communication, and learning. Large Language Models (LLMs) have shown sophisticated problem-solving and cooperative abilities, raising questions about whether they possess ToM. However, evaluations show mixed results, with some claiming human-level performance and others highlighting brittleness to minor task alterations. This ongoing disagreement stems from a lack of clear definitions and evaluation standards for ToM in LLMs.

A significant source of confusion is the definition of 'having' ToM. It can refer to: (1) matching human behavior in ToM tasks (behavior-matching), or (2) using the same mental computations or algorithms as humans (computation-matching). Behavior-matching focuses on whether an LLM's inferred mental states match human inferences, while computation-matching questions whether the underlying process is similar. For example, the classic Sally-Anne task, designed to test false belief attribution in humans, is often used as a ToM evaluation for LLMs. However, passing such a test through simple heuristics (like a lookup table) wouldn't constitute evidence of human-like computation. The article argues that the conflicting findings in LLM ToM research often depend on which definition is implicitly used: positive claims typically focus on behavior-matching, while negative claims emphasize the lack of human-like computations.

Many studies report that LLMs, particularly advanced models like GPT-4, can achieve human-level performance on classic ToM tasks, including false-belief scenarios and higher-order inferences, suggesting emergent ToM abilities. Factors like model size, fine-tuning, and few-shot prompting positively influence performance. Conversely, other research demonstrates that LLMs are brittle, failing on adversarial examples or minor modifications to tasks that humans would easily navigate. LLMs also struggle with interactive settings, social commonsense, and higher-order recursive reasoning about beliefs. These failures suggest that LLMs might rely on shallow heuristics rather than robust, generalizable ToM abilities. This mixed evidence underscores the importance of the distinction between behavior-matching and computation-matching.

Two major issues plague current ToM evaluations for LLMs. Firstly, the over-emphasis on behavior-matching leads to a focus on cataloging all possible ToM behaviors rather than understanding the underlying computations. The article advocates for computation-focused benchmarks, like the AGENT and BigToM frameworks, which assess core computational abilities and model generalization, inspired by cognitive science theories such as inverse planning. Secondly, evaluations suffer from validity threats, where models might be 'right for the wrong reasons' or 'wrong for the wrong reasons'. 'Right for the wrong reasons' includes 'training away' (models being continually updated on test data, creating an illusion of progress without fundamental computational changes) and exploiting unintended statistical associations or deeper heuristics. 'Wrong for the wrong reasons' arises when adversarial tests, designed to expose computational differences, inadvertently increase auxiliary task demands (e.g., complexity, physical reasoning), masking genuine ToM capabilities or taxing other cognitive resources.

Future research should explore the relationship between pragmatic communication and ToM in LLMs, investigating whether abilities like irony interpretation correlate with false-belief inference, as seen in humans. This could shed light on whether LLM learning paradigms foster similar co-emergence of abilities. Additionally, controlled learning experiments, varying training objectives and linguistic input, could reveal how ToM abilities emerge in LLMs. The article also suggests studying 'spontaneous ToM' in LLMs—ToM-like behaviors that arise without explicit prompting, mirroring human predispositions. Finally, mechanistic interpretability, coupled with normative cognitive models of ToM (e.g., inverse-planning, RSA), could help understand the internal computations generating LLM behavior in ToM tasks, by targeting 'cognitive' variables like beliefs and desires.

The persistent disagreement over LLMs' ToM capabilities stems from definitional ambiguities and methodological flaws. The article recommends shifting focus from behavior-matching to comparing the underlying computations used by humans and machines for mental state inferences. It also calls for clearer construct validity in evaluations, explicit consideration of auxiliary task demands, and the use of static, openly accessible models to prevent 'training away'. Adopting these recommendations can lead to more precise and valid measurements of ToM, enhancing our understanding of both artificial and human cognition.