Not Your Average Clever Hans: Latent Representation of Psychological Constructs in Large Language Models

Psychologists increasingly use large language models (LLMs) as experimental participants, but an effect in such a silicon sample supports a process theory only if the LLM's response arises causally through representing the intermediate psychological states, or constructs, before it is produced. We examine whether LLMs have stable representations of these psychological constructs in their latent neural layers. Using a multitrait-multimethod approach to examine 66 psychological constructs (e.g., psychological ownership, moral agency, and perceived goal progress) in 16 open-weight LLMs, we found that the LLMs' inner representations met construct-validity standards, were shared across model developers and sizes, and were grouped by appraisal processes. We also found that distinct manipulations of a construct activated the same representation, and that whether this activation affected the LLMs' responses depended on its input proximity to response production. We thus provide a method that enables researchers to study psychological mechanisms using silicon samples.