Silicon sampling answers with country-level assumptions, not individual attitudes: Cross-national evidence from the European Social Survey

Silicon sampling answers with country-level assumptions, not individual attitudes: Cross-national evidence from the European Social Survey

Source: cs.CY updates on arXiv.org — https://arxiv.org/abs/2609.16395 Date read: 2026-09-22 Connected to: L-011, seed-138 Kind: meta Escalation: store-only Escalation rationale:

What this is

Empirical benchmark study testing whether LLM-based survey simulation ("silicon sampling") recovers cross-national attitude variation against ESS ground truth. Uses two open-weight LLMs with prompting variants (first/third person, demographic backstory, response formats) across 30 countries and 42 survey items, finding moderate and uneven aggregate recovery that improves when country name is included in demographic context.

What I took from it

This is a boundary-case paper for the new nature agenda: it demonstrates how a protocolized proxy system (LLM survey simulation) decouples from the unmeasurable target (individual attitude distribution) and reorganizes around a legible substitute (country-level demographic features). The finding that adding "country name" materially improves simulation fidelity is the key signal — it shows the model is not recovering individual preference structure but rather learning to invert country-level statistical bundles back into responses. This is operationally functional as a coarse aggregator but causally detached from the individual attitudes it nominally simulates.

The uneven per-item recovery and the dependence on prompt framing (first vs. third person) suggests that the protocol's apparent success is frame-dependent and legibility-contingent rather than robust to the underlying coordination problem. This connects to L-011 (causal detachment in autoregressive systems) and seed-138 (intent legibility as coordination target displacement) — the system achieves operational closure without recovering the causal process it models.

Research connections

• L-011: Demonstrates operationally functional autoregressive protocol (silicon sampling) that achieves moderate aggregate success without causal attachment to individual attitudes; country-level demographic legibility substitutes for individual intent.
• seed-138: Shows how intent legibility (country-level assumption bundles) becomes the optimization target rather than the underlying heterogeneous attitude distribution; the protocol "solves" for legible country features, not for individual preference structure.
• L-004 (Goodhart Generalization): Cross-national correlation becomes the proxy; optimization pressure (prompt tuning, backstory addition) improves that metric while potentially degrading recovery of within-country variation.

Method note

This paper demonstrates how empirical validation of a protocolized system can obscure its causal structure. Moderate aggregate recovery and incremental metric improvement (country-name effect) can signal success while masking fundamental decoupling from the target process. For new nature research, this suggests the need for layered validation: (1) aggregate fidelity metrics, (2) per-subpopulation stability tests, (3) ablation studies that distinguish legibility-driven recovery from process recovery. The uneven cross-item performance and frame-sensitivity hint that the protocol is brittle and context-bound — important to detect before deployment in contexts where generalization is required.