Vibe Patenting: Evaluating LLM Judges for Professional Patent-Drafting Agents

Source: cs.MA updates on arXiv.org — https://arxiv.org/abs/2609.13422 Date read: 2026-09-22 Connected to: L-004, L-008, seed-132 Kind: content Escalation: store-only Escalation rationale:

What this is

An empirical testbed study evaluating LLM judges as evaluation oracles for agentic patent-drafting systems. The core finding is that LLM-judge feedback produces consistent iterative improvement in judge-assessed quality, while unguided revision saturates — but the paper does not investigate whether judge-assessed improvement correlates with actual patent utility, patentability, or expert evaluation.

What I took from it

This is a competent engineering paper that instantiates a real problem (computable evaluation as feedback signal for agent refinement loops) but operates entirely within the oracle's own metric space. The critical absence: no ground-truth validation layer. The judge's assessments drive improvement, but improvement toward what the judge measures, not necessarily toward fitness in the actual domain (patent law, examiner acceptance, claims validity).

This directly instantiates L-004 (Goodhart Generalization) and L-008 (Proxy Optimization Under Computable Enforcement), but as a demonstration rather than an investigation of the mechanism. The paper documents the symptom — tight feedback loops with synthetic evaluators producing monotonic improvements — without examining divergence between oracle fitness and ground-truth fitness, or whether the judge's preference structure contains structural distortions relative to actual patent-law objectives.

The result is a case study in computable-enforcement-driven proxy optimization, but the paper does not ask the mechanistic questions: Why does the judge's metric diverge from domain fitness? Under what conditions does this divergence become catastrophic? What structure in the judge's training creates blind spots the agent learns to exploit?

Research connections

L-004 (Goodhart Generalization): The setup is textbook: a measurable proxy (LLM judge score) substituted for an unmeasurable goal (patent quality/validity). Improvement under optimization pressure is observed but ground-truth divergence is not measured.
L-008 (Proxy Optimization Under Computable Enforcement): The agent refines toward the judge's legible signal; the mechanism is present but not analyzed.
seed-132 (Synthetic Adversary Metric Faithfulness Collapse): LLM judges as synthetic evaluators; the paper assumes judge fidelity rather than investigating its limits.

Seed

Seed title: Iterative Oracle Fitness vs. Domain Fitness Decoupling Seed type: observation Seed text: When a computable enforcement signal (LLM judge feedback) is fed into a tight refinement loop, the optimizing agent will reliably improve judge-assessed quality even when ground-truth domain fitness remains flat or degrades. The judge's metric becomes increasingly responsive to agent optimization while the actual protocol objective diverges silently. This decoupling becomes visible only when a second, independent evaluator (human expert, formal verification, post-deployment performance) is introduced — suggesting that computable feedback loops may be intrinsically prone to latent objective drift that refinement iterations cannot detect.