Privileged Self-Distillation

Introduction

Large language models are increasingly used for agentic tasks such as coding, research, and knowledge work, where success depends on carrying out a sequence of decisions over time. Post-training has driven much of this shift by teaching models to follow instructions, solve multi-step problems, and use tools effectively.

Post-training methods can be categorized across two axes:

• Where do the training states come from? on-policy vs off-policy
• How is that feedback used as supervision? outcome signal vs constructed signal

Off-policy methods train on data generated outside the student policy. A common example is supervised fine-tuning (SFT) on teacher trajectories or fixed datasets. These sources provide clean token-level targets, but they have a coverage limitation: states selected by the external source may not include the mistakes the student encounters in its own rollouts.

On-policy methods address this coverage gap by drawing training states from the student's own rollouts. In RLVR, a verifier scores each attempt. One common algorithm is group relative policy optimization (GRPO), which compares rewards within each sampled group.

Consider a simple scheduling task with two sampled rollouts.

The verifier marks A as failed and B as successful, so GRPO favors rollouts like B over rollouts like A. This is useful because the policy learns directly from the outcomes of its own attempts. But the signal is coarse: each rollout receives only one binary outcome for the entire trajectory, telling us which attempt worked but not that the AM/PM choice mattered. A second limitation appears when both rollouts set 8 AM: their identical rewards leave the group with zero reward variance and no learning signal.

On-policy distillation (OPD) provides a denser alternative by replacing the single outcome with local teacher targets. A stronger teacher is queried at the same prefixes the student visited, providing dense token-level supervision without requiring reward variation. In the scheduling example, the teacher scores each decision in the failed rollout.

The dense targets reveal where the teacher disagrees: opening the calendar receives little correction, while setting 8 AM receives the main correction as the teacher shifts probability toward 8 PM. OPD has a structural limitation: after a wrong choice enters the trajectory, later teacher queries remain conditioned on the history it creates. Confirming 8 AM is wrong for the task but locally coherent once 8 AM is already in the history, so it receives almost no correction.

This is prefix drift: after an early mistake, later teacher queries are conditioned on the history that mistake created. Those targets can still teach useful recovery behavior, but they describe continuations of the student's branch rather than a corrected trajectory.

Cursor's Composer 2.5 writeup addresses this by using hinted distillation to give the teacher extra context at the mistaken decision. Place a short hint before the target message, query the teacher under that privileged context, and train the unhinted student toward the resulting distribution. Here, the hint is a reminder to check AM/PM, while the student remains unhinted.

Hinted distillation shows how to transfer a correction, but it does not tell us what hint to use or where to place it. Think of a teacher correcting a student's mistake: the goal is to reveal the misconception without solving the problem for them. An overly specific hint may fix the problem in front of the student without teaching a reusable strategy. A vague hint may not explain what went wrong. The useful hint lies between these extremes and must arrive while the trajectory can still recover.

A failed on-policy trajectory offers the raw material for repair: the student's actions, the environment's responses, and the prefixes where recovery may still be possible. The challenge is to construct an intervention that fixes the trajectory and yields a target the unhinted policy can learn. We address this challenge with PSD.

Privileged Self-Distillation

Privileged Self-Distillation is a post-training method that uses evidence from a failed rollout to construct a verified local training target, then distills that target into the unhinted policy. It proceeds in two stages: verified hint construction and local self-distillation.

Verified hint construction. Given a failed rollout, a hint constructor reads the completed trace, selects a student-reached prefix, and proposes a short hint. PSD reruns the policy from that prefix with the hint and retains the intervention only if the continuation passes the verifier.

Each accepted intervention produces a distillation target with three properties:

• Local, because it is anchored to a prefix the student reached and the repair loss is applied only at the selected decision.
• Verified, because the hinted continuation passes the verifier.
• Teachable, because the hint is short, procedural, and screened for answer leakage.

Local self-distillation. Each accepted hint creates two views of the same student-reached state. The teacher, a frozen snapshot of the round-start policy, sees the original prefix plus the hint. The student sees the original prefix alone and is trained toward the teacher's hinted next-message distribution only at the selected decision.

Training

Before distillation, PSD must determine which proposed repairs are reliable enough to learn from.

A PSD round has four steps:

1. Sample on-policy rollouts. Collect fresh trajectories from the current policy.
2. Construct and verify local repairs. For each failed rollout, propose hints at candidate decision points, rerun the policy from each selected prefix, and retain repairs that pass the verifier.
3. Compute local distillation targets. Query the frozen policy with each verified hint and construct unhinted preservation targets from passing turns.
4. Optimize the policy. Train with equally weighted repair and preservation losses.

policy = initial_checkpoint for round in rounds:    # 1. Sample on-policy rollouts    frozen = freeze(policy)    rollouts = sample_rollouts(frozen)     # 2. Construct and verify local repairs    verified_repairs = []    for rollout in failures(rollouts):        for step, hint in hint_constructor(rollout):            prefix = rollout.prefix_before(step)            continuation = continue_from(frozen, prefix, hint)             if verifier(continuation).passes:                verified_repairs.append((prefix, hint))     # 3. Compute local distillation targets    repair_targets = local_hinted_targets(        verified_repairs, teacher=frozen    )    preservation_targets = unhinted_targets(        collect_passing_turns(frozen), teacher=frozen    )     # 4. Optimize the policy    policy = optimize(        initial_policy=frozen,        repair_targets=repair_targets,        preservation_targets=preservation_targets,        loss_weights=(1.0, 1.0),    )

Each accepted repair is a pair $(t, h)$, where $t$ denotes the selected assistant turn and $h$ the verified hint. Let $c_t$ denote the original rollout prefix before that turn, and let $\widetilde{c}_t = \operatorname{insert}(c_t, h)$ denote the same prefix with the hint inserted. The frozen round-start policy $\pi_{\bar\theta}$ acts as the teacher, while the trainable policy $\pi_\theta$, initialized from the same checkpoint, acts as the student:

The repair objective is evaluated tokenwise only over the selected assistant turn, not the full rollout.

Preservation targets. To prevent repair training from forgetting behavior the policy already performs correctly, we collect assistant turns from passing rollouts generated by the teacher. For each original, unhinted context $c$, the teacher and student condition on the same context:

We approximate each token-level KL using the teacher's top-20 tokens and weight the repair and preservation loss terms equally.

The verifier only determines which repair candidates enter training. For accepted repairs, the learning signal comes from the hinted teacher distribution, not the verifier outcome.

PSD can use different procedures to select a repair point and propose a hint. We study self-hinting, where the student diagnoses its own failures, alongside hint construction by a stronger frontier model. Our self-hinting procedure uses Qwen3.5-9B with a harness that constructs hints from the student's own failed trace. We optimized the harness to select a recoverable decision point and produce a minimal hint that enables a verifier-passing rerun.

In the frontier setting, Opus 4.8 runs in a Claude Code agent that analyzes the student's failed trace and searches for a verified repair. Within a fixed retry budget, it selects a repair point, proposes a hint, reruns the task, and revises the hint until the continuation passes the verifier. Opus is used only for repair construction; Qwen3.5-9B supplies both the hinted teacher and unhinted student distributions used for training.

A failed rollout is raw experience, not yet supervision. Different post-training methods turn it into different training objects.

Results

We compare several post-training methods for Qwen3.5-9B, using 360 BFCL tasks for training at comparable budgets and a separate 240-task held-out suite for evaluation.

On our held-out BFCL split, the Qwen3.5-9B baseline passes 59 of 240 tasks. Verified-retry SFT reaches 77.2, while OPD reaches 80. Self-generated hints, written from the 9B policy's own failed traces without an answer key or external model, reach 86.3, above OPD at this budget. Single-pass hints from the stronger Opus 4.8 model reach slightly higher, at 88.0. Our agentic PSD setup runs Opus 4.8 in a Claude Code harness and reaches 105.7.

Tasks become invisible to group-relative RL when every sampled rollout fails. In the first collection, 183 of 360 tasks had this pattern, so they provided GRPO no group-relative signal.

Coverage varies by method.

Agentic PSD turned 141 of the 183 all-fail tasks into verified repairs. That supervision was useful on its own: corrective examples from this slice, alongside preservation, raised held-out performance from 59 to roughly 104.

More importantly, repair shrank the dead zone. Across three rounds, PSD reduced all-fail tasks from 183 to 43 and reached 118.0. GRPO remained near 140 all-fail tasks and reached 111.3.

We test whether verified repair can serve as a warm start for GRPO.

Here we compare three three-round training paths from the same base model: GRPO for all three rounds (blue), PSD for all three rounds (black), and one PSD warm-start round followed by two GRPO rounds (green).

The repair checkpoint reduced all-fail groups from 183 to 47. After two GRPO rounds, the warm-started path reached 146.5, compared with 118.0 after three PSD rounds and 111.3 after three GRPO rounds.

Relation to recent work

Recent work supplies pieces of this picture. HINT-SD targets failure-relevant spans rather than distilling everywhere. OPSD and π-Distill condition a self-teacher on privileged solutions or frontier traces. Pedagogical RL trains a privileged self-teacher to generate complete trajectories that are both successful and learnable. PSD uses a different unit of supervision: a local repair constructed from the student's failed trace and accepted only after it recovers the task on rerun.

What remains

Writing a minimal, teachable hint from a failed trace is something the student model can already partly do: its own single-pass hints reach 86.3 against 88.0 for a frontier writer. The agentic tier is where the gap remains: our frontier agent uses Opus 4.8. If agentic self-diagnosis became a native capability, PSD would need no external model at all: a policy could improve from its own mistakes with only a verifier.

BFCL is unusually friendly to repair: states can be replayed from a local prefix and success checked automatically. Long-horizon environments where state cannot be restored cheaply would require checkpointing or another way to verify counterfactual repairs.

What this suggests

PSD rests on a simple idea: failed rollouts can become supervision if we find and verify the intervention that would have made them recover. In our experiments, PSD with self-generated hints scored higher than path distillation at a comparable budget, while Agentic PSD constructed verified supervision for 141 of GRPO's 183 first-round all-fail groups. Across three rounds, standalone PSD scored higher than GRPO, and one PSD warm-start round followed by two GRPO rounds produced our strongest result.

The broader lesson is that post-training depends not only on how behavior is scored, but on how the context that makes behavior teachable is constructed. PSD treats that context as something a hint constructor can search for, the environment can verify, and the policy can learn to act without.

Citation

Please cite this work as:

Essam Sleiman, "Privileged Self-Distillation", July 2026. https://canvas.inc/research/privileged-self-distillation

Or use the BibTeX citation:

@misc{sleiman2026psd,
  author = {Essam Sleiman},
  title = {Privileged Self-Distillation},
  year = {2026},
  note = {https://canvas.inc/research/privileged-self-distillation},
}