Human-AI Collaboration for Estimating Scientific Replicability

Source: cs.MA updates on arXiv.org — https://arxiv.org/abs/2605.27394 Date read: 2026-05-29 Connected to: L-004 Escalation: store-only Escalation rationale:

What this is

A methodological paper proposing a hybrid human-AI system for assessing scientific replicability, treating replicability as a forecasting task that combines expert judgment with learned models. The work addresses limitations in purely human or purely automated approaches but does not present a sustained theoretical argument about protocol behavior or formalize a mechanism absent from the current inventory.

What I took from it

This paper illustrates L-004 (Goodhart Generalization: Metric Capture) in applied form, but does not advance understanding of why or when metric capture occurs in protocolized systems. The core problem—that replicability is unmeasurable in advance, forcing reliance on proxy indicators (methodology rigor, author track record, statistical power estimates)—is L-004's domain. However, the paper's response (combining human and machine forecasts) is a mitigation strategy, not an investigation of the law itself. It confirms that the measurement problem is hard, not that it obeys a generalizable protocol-level regularity.

The work touches on H-002 (trust accumulation as function of age/stability) insofar as human experts may weight historical publication patterns, but this is peripheral and not tested.

Research connections

L-004: Replicability assessment necessarily proxies an unmeasurable ground truth; the paper acknowledges but does not theorize the capture dynamics that follow.
H-002: Human forecasters may implicitly weight protocol age/institutional stability, but this is not isolated or measured in the work.

Candidate laws or signals

None. This is a tool paper addressing a domain-specific forecasting problem, not a claim about how protocolized systems behave under scaling, adoption, or formalization pressure.