Human-in-the-Loop Large Language Model Framework for Identification of Cutaneous Immune-Related Adverse Events

Shallow read  ·  2026  ·  source  ·  all reading

L-004 L-012

Human-in-the-Loop Large Language Model Framework for Identification of Cutaneous Immune-Related Adverse Events

Source: cs.MA updates on arXiv.org — https://arxiv.org/abs/2607.20428 Date read: 2026-09-02 Connected to: L-004, L-012 Kind: meta Escalation: store-only Escalation rationale:

What this is

A clinical decision-support tool paper demonstrating LLM-assisted workflow for adverse event detection in medical notes. Reports performance gains (F1: 0.77→0.88, kappa: 0.50→0.82, 50% time reduction) using retrieval-augmented multi-agent LLM with human review loop.

What I took from it

This is a benchmark/application paper, not a primary theoretical or empirical investigation of protocol dynamics. The "human-in-the-loop" framing obscures the actual mechanism: the LLM becomes a legible intermediate layer (extracting structured adverse-event signals from unstructured text), which displaces where human judgment is applied — from holistic clinical reasoning to validation of machine-legible classifications. The kappa improvement (0.50→0.82) likely reflects that humans now agree on machine-extracted signals rather than on clinical judgment itself; this is consistent with L-012 (intervention-layer displacement), but the paper does not investigate whether this displacement introduces novel failure modes (e.g., convergent misses, paradigm-locked tolerance of systematic exclusions). The time reduction is presented as pure efficiency gain, but it may reflect task simplification rather than capability enhancement — humans are no longer doing the hard work of recognizing the event, only confirming a pre-classified candidate. This is exactly where proxy optimization (L-004) becomes dangerous in safety-critical domains, but the paper has no mechanism analysis.

Research connections

  • L-004: The LLM output becomes a measurable proxy for "adverse event present"; under deployment pressure, optimization will target F1 rather than clinical validity, but no mechanism is studied here.
  • L-012: Human review is displaced from signal detection to signal validation; the locus of error moves from human miss to machine hallucination + human confirmation bias, but this is not characterized.
  • seed-062 (Formalization Opacity Collapse): Clinical reasoning (opaque) → LLM extraction (legible) → human validation (pseudo-legible); unclear whether legibility gain is real or illusory.
  • seed-073 (Correlated Failure Under Proxy Consensus): If multiple raters now agree via machine-legible intermediate, failure modes become correlated; not explored.

Method note

This paper exemplifies a gap in AI-in-critical-domains research: performance metrics (F1, kappa, time) are reported without characterizing what changed about the task itself when automation was introduced. The kappa improvement may indicate genuine coordination gain or may indicate that humans are now simply agreeing on machine outputs rather than independent judgments — these are mechanistically opposite. Future work should separate (a) does the system detect events humans miss? (b) do humans collude with the system's errors? (c) are there systematic classes of events the system is blind to? Shallow metrics obscure protocol-level risk.