Unsafe at any AUC: Unlearned Lessons from Sociotechnical Disasters for Responsible AI

Shallow read  ·  2026  ·  source  ·  all reading

L-004 L-013

Unsafe at any AUC: Unlearned Lessons from Sociotechnical Disasters for Responsible AI

Source: cs.CY updates on arXiv.org — https://arxiv.org/abs/2607.14353 Date read: 2026-09-02 Connected to: L-004, L-013 Kind: content Escalation: store-only Escalation rationale:

What this is

A sociotechnical systems analysis applying historical catastrophe studies to AI governance and risk. The work uses documented human-made disasters (Three Mile Island, Challenger, etc.) as case material to argue that "responsible AI" frameworks fail because they isolate technical components from the institutional, incentive, and coordination structures that produce failure at systems level.

What I took from it

The paper reinforces L-013 (Paradigm-Locked Anomaly Tolerance) by showing how organizations systematize the suppression of early warning signals — not through incompetence, but through structural rationality: safety metrics become decoupled from actual risk when the cost of acknowledgment exceeds the perceived benefit of intervention. The work confirms that metric capture (L-004) operates not just on optimization surfaces but on institutional attention allocation: AUC scores, fairness coefficients, and deployment safety checklists become the protocol's official model of risk, crowding out qualitative anomaly detection.

However, the paper does not isolate a new mechanism in the protocol layer. It restates the sociotechnical failure mode (institutions ignore signals that contradict their operating model) in AI context. It does not explain why this pattern persists in some protocol architectures and not others, or what structural properties make a system resistant or vulnerable to this form of ossification. It reads as a policy/design guidance paper rather than a primary source proposing a testable regularity.

Research connections

  • L-004: Confirms metric capture creates institutional blindness; does not isolate new mechanism.
  • L-013: Historical case material for paradigm-locked anomaly tolerance in safety protocols; applicable but not novel to the inventory.
  • seed-073 (Correlated Failure Under Proxy Consensus): Marginal: the paper suggests that consensus around a single safety metric can create correlated institutional failure modes, but does not formalize the mechanism.

Seed

Seed title: none

Seed type: —

Seed text: —