Constructive Alignment: Governing Preference Dynamics in Human-AI Interaction

Source: cs.CY updates on arXiv.org — https://arxiv.org/abs/2607.00001 Date read: 2026-09-01 Connected to: L-004, L-008 Kind: content Escalation: store-only Escalation rationale:

What this is

A position paper reframing AI alignment from preference inference to preference construction. The work argues that human preferences are dynamic and co-shaped through interaction with adaptive systems, and proposes "constructive alignment" as a control framework. Primary domain: human-AI interaction design and alignment methodology.

What I took from it

The paper directly engages L-004 (Goodhart Generalization) and L-008 (Proxy Optimization Under Computable Enforcement) by adding a temporal and relational layer: the metric system itself (the AI's proxy for human preference) doesn't just corrupt the target—it regenerates what counts as the target through repeated interaction. This is a mechanism-level contribution to L-008: when enforcement signals become legible and optimization is continuous and personalized, the system doesn't just optimize the proxy harder; it actively shapes what the agent comes to value.

However, the paper remains primarily a normative reframing and design proposal rather than a sustained empirical or theoretical account of when and how this construction happens, or under what conditions it stabilizes vs. spirals. The "control problem" framing is suggestive but underdeveloped—it names the phenomenon without mechanistic prediction power. The work is conceptually clean but doesn't yet provide the evidence density or formal structure needed to ground a new law or settle open mechanisms in L-008.

Research connections

L-004: Extends Goodhart by adding that metrics shape preferences over time, not just distort fixed ones; confirmation in principle, but without the causal structure or failure conditions.
L-008: Directly relevant—shows a potential mechanism for why proxy optimization under computable enforcement becomes self-reinforcing, but lacks specificity on equilibrium conditions and reversibility.
seed-012 (Intervention-Layer Displacement): Related pattern—the locus of optimization pressure shifts when a preference becomes legible and actionable; this paper treats that as design problem rather than protocol law.
seed-004 (Goodhart Generalization): Constructive alignment suggests Goodhart operates not as a one-time distortion but as a cycle—metric capture regenerates the target itself.

Seed

Seed title: Metric-Induced Preference Ratcheting in Adaptive Systems

Seed type: motif

Seed text: In personalized adaptive systems where optimization signals are continuously legible and enforcement is continuous, the system does not merely optimize a fixed proxy for an unmeasurable goal—it progressively reconstructs what counts as the goal itself through repeated feedback loops. Over sufficient interaction time, the agent's revealed preferences converge toward an artifact of the metric rather than recovering any independent underlying preference. This ratchet is asymmetric: preferences shaped by the metric are harder to distinguish from "authentic" preference than the metric is to remove, creating path dependence in value formation. The mechanism generalizes beyond preference-learning to any adaptive protocol where the target property is initially unmeasurable and a computable proxy must be used as a stand-in during deployment.