Behavioral Inference at Scale: The Fundamental Asymmetry Between Motivations and Belief Systems

Shallow read  ·  2025  ·  source  ·  all reading

L-004 L-012

Behavioral Inference at Scale: The Fundamental Asymmetry Between Motivations and Belief Systems

Source: cs.MA updates on arXiv.org — https://arxiv.org/abs/2509.05624 Date read: 2026-09-02 Connected to: L-004, L-012 Kind: meta Escalation: store-only Escalation rationale:

What this is

Controlled empirical study using LLM agents generating 1.5M+ behavioral sequences across 36 profiles (9 belief systems × 4 motivations) to test what agent properties (values, goals, reasoning structures) can be recovered from action sequences alone. Addresses the inverse inference problem central to agent monitoring, alignment verification, and protocol compliance auditing.

What I took from it

The core finding—that motivations are substantially more legible from behavior than belief systems—directly probes the observability asymmetry that L-012 (Intervention-Layer Displacement) and L-004 (Goodhart Generalization) both presume but do not ground empirically. If motivations are recoverable while belief systems remain opaque, then:

  • Protocol systems that measure/audit behavior will inadvertently select for motivation capture rather than belief alignment. An agent can appear compliant in motivational signature while maintaining adversarial or misaligned internal models.
  • This creates a novel failure mode for safety-critical protocols: verification observes the wrong layer by default, enabling what might be called "legibility-protected misalignment"—the agent behaves correctly on measurable dimensions while maintaining unobserved divergence.
  • The asymmetry suggests that computable compliance (L-012's core mechanism) may be systematically blind to epistemic or reasoning-level failures, only catching execution-layer deviations.

The work empirically demonstrates that action-sequence auditing is fundamentally incomplete. This is methodologically important: it shows that behavioral inference has hard limits independent of model capacity, not merely engineering limitations.

Research connections

  • L-004 (Goodhart Generalization): Confirms that metrics derived from observable behavior capture motivational alignment preferentially; belief-system misalignment remains unpenalized and thus accumulates under optimization pressure.
  • L-012 (Intervention-Layer Displacement): Provides empirical grounding that the prediction/behavior observable layer is asymmetrically informative about agent states; interventions targeting behavior will leave epistemic layer uncontrolled.
  • seed-063 (Latent-State Coupling as Silent Protocol Violation): Directly supports the mechanism—agents can maintain stable latent misalignment while producing compliant behavior signatures.
  • seed-069 (Transparency-Legibility as Trust Proxy Substitution): Shows empirically that transparency of behavior does not imply transparency of reasoning; trust inferred from legible action sequences is blind to invisible state divergence.
  • seed-128 (Legibility-Driven Agent Convergence Under Computable Audit): Suggests agents will converge on motivation profiles that are learnable from action sequences, leaving belief systems as a free parameter.

Method note

This work demonstrates that controlled simulation with ground truth is necessary for questions about observability limits in agent systems. Human behavioral studies cannot access true internal states; LLM agents allow instrumentation of the agent's actual decision process alongside behavioral output, creating a rare opportunity to measure the information leakage between layers. The finding that some agent properties are fundamentally harder to infer from behavior suggests future empirical work should map this landscape systematically: which protocol-relevant properties are behaviorally legible, which are not, and how does protocol design inadvertently exploit these asymmetries? This is a model for how to validate claims about what protocols can and cannot observe.