Publication bias and p-hacking in the effect of COVID-19 on learning

Source: arXiv:2608.00580v1 — https://arxiv.org/abs/2608.00580 Date read: 2026-09-02 Connected to: L-004, L-013 Kind: meta Escalation: store-only Escalation rationale:

What this is

A meta-analytic audit of pandemic learning-loss estimates using multiple bias-correction techniques (PET-PEESE, 3PSM, RoBMA) to assess whether the field's consensus estimate reflects genuine effect or selective publication and methodological fishing. This is methodological meta-research, not a primary theoretical or empirical investigation of protocolized systems.

What I took from it

This work exemplifies L-013 (paradigm-locked anomaly tolerance) in institutional operation: a field converges on a central policy-relevant estimate without systematic interrogation of whether publication filters and researcher incentives have inflated it. The paper's apparatus—multiple correction methods applied to the same corpus—shows how a research protocol (peer review + publication economics) can ossify around a metric without triggering internal audit.

It also illustrates L-004 (Goodhart generalization) in applied research: standardized effect sizes become the measurable proxy for "learning impact," but under publication pressure and methodological flexibility, the proxy itself becomes the optimization target rather than the unmeasurable underlying phenomenon (actual human-capital loss). The correlation between effect size and standard error becomes itself a legible optimization surface.

However, this paper is itself a meta-level intervention—a correction protocol applied to a protocol. It does not present a sustained theory of how protocolized systems behave; it audits one instance of bias accumulation. The findings are important for policy but not generative of new mechanism.

Research connections

L-004: Standardized effect size as measurable proxy for unmeasurable learning loss; publication pressure drives metric capture, not measurement of true construct.
L-013: Established educational-economic research protocols tolerate accumulating evidence of bias (p-hacking, publication selection) without triggering systematic re-examination until external audit applied.
seed-073: Correlated failure under proxy consensus — field consensus around a single pooled estimate masks underlying distribution of selection bias across studies.

Method note

This meta-research suggests that fields employing legible, quantitative proxies (standardized effect sizes, p-values, confidence intervals) require built-in audit protocols rather than treating meta-analysis as post-hoc correction. The existence of multiple bias-correction methods (PET-PEESE, RoBMA, 3PSM) indicates no stable ground truth—suggesting the protocol itself (how estimates are published, selected, and pooled) is the unit that requires redesign. Audit trails and transparency alone do not prevent metric capture; the economic and epistemic incentives that drove the capture must be addressed in the protocol layer.