Borrowing a Foothold: Distilling Simulated Annealing into a 9B Agent

The goal is to hill-climb performance on FrontierCS optimisation problems by teaching the 9B a usable algorithmic skill. I want it to construct an executable simulated annealer or local search, then improve that solver through tests and submission feedback. Part 3 ended with the 27B already writing the kind of search the 9B rarely made work. A teacher was the next route to that missing foothold. The transfer I attempted included choosing a representation, implementing the search, testing it, repairing it and keeping the best result across a long episode.

This is a harder transfer than copying a final answer. The teacher chooses the states it visits, the size of its programs and how long to think before acting. Those choices determine what the student will see in training. Generalized Knowledge Distillation treats the mismatch between teacher-generated training sequences and student-generated inference sequences as a central problem. A long tool-using episode gives that mismatch many chances to compound.

I started with 612 teacher attempts on 153 training problems, four per problem, with a 90-minute deadline and a roughly 57K-token thinking budget per call. These were long tool-using episodes: think, edit, compile, test, submit, read feedback, and continue across context compactions. Compaction replaces older history with a summary and retained turns; the student must then carry on with the program and its constraints intact. The student started from the earlier single-call RL checkpoint, then received one epoch of off-policy supervised fine-tuning on those teacher-generated traces. There was no replay of the 9B’s old trajectories and no outcome-quality filter on whole attempts. The builder still excluded flagged turns, including forced endings and compile-error-only turns. Thinking remained in the loss on trained turns, along with actions and compaction summaries.

The 27B reference scored 23.50 on the 40-problem agent evaluation, four attempts per problem. The student had a large regression instead. Under the frozen protocol, Avg@4 fell from 5.99 to 0.53, a paired difference of −5.46 points [−9.04, −2.42]. A halfway checkpoint scored 0.64. This was not a case where the final few updates spoiled an otherwise useful run.

Before fine-tuning, the student submitted early and iterated: about 27 judged submissions per attempt, with the first arriving at a median of 1.2 minutes. Afterwards it averaged 0.6 submissions. Among attempts that submitted, the first arrived at 132 minutes. Seventy-four percent submitted nothing, and 94% earned zero credit. The model spent its time producing long thinking, repeatedly filling the context, and trying to continue.

Analysis · The behaviour can transfer before the algorithm

A teacher can afford a long plan if it can turn that plan into a strong program. For this student, the same delay consumed the time that previously paid for tests and revisions. That creates a plausible feedback loop: late code means less execution feedback, so an unreliable first construction has fewer chances to recover. The submission statistics support the loss of that opportunity; they do not tell me which part of the teacher’s reasoning caused it.

The full episode bundled algorithm construction, pacing, tool use and memory across compactions into one imitation target. A regression on that bundle cannot identify which prerequisite was missing. The curriculum needs to test the first working algorithm before restoring the full clock. A matched experiment that preserved early tested code while transferring search would weaken my pacing explanation.

A bad result is easy to explain with a suspected training bug. Here the later SFT audit checked the accounting directly. An independent forward pass reproduced the trainer’s first-batch loss to about 0.000014 nats per token. Planned and trained token counts matched on all 77 steps. The evaluated export was the intended checkpoint, and the prompt and tool-round templates matched the evaluation. These checks narrow the failure; they do not certify every long-sequence backward pass.

What the audit exposed was the composition of the objective. The corpus contained about 204 million loss tokens; 70.6% were thinking. On the first training batch, thinking accounted for 92.7% of initial loss mass, because it was both abundant and much less predictable to the student. The student already assigned relatively high likelihood to the teacher’s visible actions.

Roughly 90% of the total NLL reduction came from thinking. On the final turn-ending token, likelihood actually got worse. These are in-sample likelihood measurements; they do not measure algorithmic capability or directly explain when the student stops at inference.

Analysis · Predictable code is not yet usable code

Low NLL on visible code does not establish that the student can independently assemble the same solver. Here every next token was predicted with the teacher’s preceding tokens and tool history available. At inference, the student has to produce that prefix itself. A small mistake in a score delta, a move operator or a stopping condition can break the algorithm even when most code tokens are unsurprising. Token likelihood and executable capability answer different questions.

Nor was the problem simply that a few enormous episodes overwhelmed all the others. The longest 10% of traces received 13.9% of total SFT weight, versus 10% under equal weighting per trace. The much larger skew was inside each trace, toward thinking. That points toward measuring which verified constructions and improvements the objective teaches. It does not yet justify removing reasoning: a controlled curriculum can retain reasoning while asking for a working program as its first measurable capability.

On-policy distillation is attractive here because it moves supervision into the student’s own trajectory. The student samples; the teacher scores what the student actually wrote. Thinking Machines’ implementation uses a sampled-token reverse-KL signal. In its simplest full-support form, the detached per-token advantage is:

A token the teacher finds relatively more plausible gets a positive local signal. This is dense supervision, but it still asks the student to approach the teacher’s conditional distribution. It does not directly say whether a completed search program works.

The early runs developed a familiar failure: longer planning, fewer calls and almost no submissions. A later arm combined a decaying teacher term with outcome reward after the whole-trace SFT. It also ran away in length. The audit, however, found real confounds that prevent a clean method verdict:

• The two probabilities initially described different distributions. The student term came from a sampler with truncation and penalties, while the teacher term was raw. That made part of the signal a sampling artefact. It was fixed before the first nonzero-learning-rate update, but the earlier gradient still mattered through the optimizer state.
• A zero-learning-rate step changed Adam’s memory. It left the parameters fixed while updating the moments. The audit attributed about 47% of the second step’s first-moment direction to that first, faulty gradient. A parameter no-op was not an optimizer no-op.
• Training and serving used different rounded weights. Training rounded the full-precision master weights to nearest; serving used stochastic rounding. With very small updates, those were different policies. Their token-probability gap grew above its initial numerical floor.

The weight mismatch was concrete enough to test. In the companion outcome-RL arm, making the trainer use the same stochastically rounded weights as serving reduced the mean log-probability gap from 0.0257 to 0.0095, close to the roughly 0.008 starting floor. That supports the numerical diagnosis. It does not establish that fixing it would make the teacher’s algorithms transfer.

There was a second reason to distrust a convenient explanation. A high share of negative teacher signals at the thinking-stop token initially looked like direct pressure never to stop. The audit found that the median absolute signal there was only 0.0007 nats, and many stop positions had zero gradient under the truncated sampler. Sign counts alone had exaggerated the story.

The verdict is limited: these runs failed to deliver the desired behaviour, and one later arm was confounded by implementation errors. They are not evidence that on-policy distillation cannot work at this size. The clean SFT regression and the confounded distillation arm belong in different columns.

Analysis · Dense supervision still needs a useful target

Moving to student-generated states addresses one distribution mismatch. It does not make the teacher’s next-token preference a test of algorithmic correctness. If the student rarely reaches a valid annealer, a dense signal can change how it describes or plans a search before it produces one. My next test is therefore OPD on the narrower construction task, with independently checked programs and a matched sampling distribution. Improvement in verified algorithm pass@k would be evidence of the missing foothold; a lower token-level divergence alone would not.

The companion arm skipped the teacher term and applied outcome RL to the same SFT checkpoint. Its credited training reward rose from about 0.08 early in the run to 0.257 at step 13. Median submissions rose from zero to four. By the later steps, the model was much more likely to submit a program and stop on its own.

The traces showed a recovery toward short, greedy programs. In a within-group check, the best search beat the best greedy program in only 7 of 40 early groups and 8 of 46 later groups. These are training diagnostics, not held-out algorithm scores, but they explain the pressure the reward supplied: search still usually lost.

The step-5 evaluation showed no significant improvement over the SFT start: −0.63 points [−2.31, +0.68]. The step-10 evaluation was incomplete, so I do not treat its provisional positive point estimate as a result. The run ended at step 13 when I moved to the takeover experiment.

Analysis · The reward can favour the skill the student already has

If a fragile search usually loses to a short greedy solver within the sampled group, the outcome objective has little reason to preserve that search. Recovering submissions can then be useful progress while moving the policy toward its existing competence. This is a candidate mechanism, supported by the within-group comparisons, rather than proof that RL cannot discover a better algorithm.

The curriculum changes that starting contest. First make correct search constructions frequent enough to evaluate; then ask whether their tested revisions beat the starting program. I would measure a gain within the same problem and program lineage, alongside held-out from-scratch performance. That separates learning to improve a solver from getting a more favourable draw of tasks.

If whole traces imported too much, perhaps a teacher could supply only the missing move. This follows the motivation behind DAgger: ask for expert guidance in states the learner actually reaches. The experiment here was a probe of whether that guidance creates useful continuations. It was not yet a completed DAgger training loop.

I saved 66 states from the earlier RL-trained 9B, half selected at random call boundaries and half at behavioural pivots: first submission, a plateau or loop onset. The planned post-compaction quota was reassigned because this student did not compact in the collection run. Each state branched three ways: the 9B continued alone; the 27B took up to three calls; or it took up to eight. A takeover ended earlier after its first completed judged submission. Then the 9B resumed, with up to 45 minutes of continuation. That made 198 forks.

The teacher received the student’s exact full context, including its earlier thinking. Teacher generation time was charged using a measured model of the student’s latency. To avoid paying for lucky repeated judging, the primary comparison re-graded each arm’s best-claimed submitted program three times. The paired intervals were clustered by problem, since several states could come from the same problem.

Three calls changed the score by −0.5 points [−2.0, +0.9]. Eight changed it by +1.0 [−0.4, +2.3]. Neither passed the pre-registered score gate, so takeover SFT did not proceed as designed.

The trace reads were more interesting than a blanket “the student throws everything away.” With the student’s history in view, the teacher often adopted its short planning and quick-edit rhythm. After hand-back, individual student continuations rewrote the whole file and quit. Across the completed readout, though, the separate code-retention gate passed. Some teacher search survived; it too rarely produced a better score.

Analysis · Keeping the teacher’s code is only one prerequisite

The retention result rules out a convenient blanket explanation: every failure was not the student immediately discarding the teacher’s work. A retained program may still have the wrong representation, a weak neighbourhood or a parameter setting the student cannot improve. Conversely, copying the student’s quick-edit style may make the teacher’s intervention too small to supply the missing algorithm. This probe did not isolate those possibilities.

The stage-two curriculum should therefore record the starting program’s verified capability, the edit, the check and the resulting gain separately. It should also include student-generated starting programs so that seeded success does not hide a construction failure. This takeover probe supplies evidence about short continuations; because its collected states did not follow compaction, it leaves post-compaction recovery untested.

Another attempt reduced the teacher’s per-call thinking budget from about 57K to 16K. The hope was to generate demonstrations closer to the student’s pace while keeping the teacher’s search skill.

The teacher often continued the same planning across calls. In the pre-flight of 58 attempts, 76% of planning calls were cut and 35% of attempts chained cuts. Thinking before the first program fell from a median of roughly 45K tokens to 33K, rather than to 16K. A call boundary had changed more than the underlying work.

The pre-flight retained enough search to continue, but it also changed the harness and environment, so it was not a clean budget-only comparison. Generation later stopped before a new SFT result. I cannot claim that short-budget demonstrations solve the transfer problem, or that this probe ruled them out.

Analysis · A token cap is not a curriculum

The teacher could move unfinished deliberation into the next call without simplifying the algorithm it was trying to build. Shorter calls therefore need not give the student shorter dependencies or an earlier executable foothold. The relevant measures are total thinking before the first verified program, time left for improvement and the quality of that program. A useful short-demonstration recipe must improve those episode-level quantities while retaining working search.

Long episodes make it easy to collect more tokens and harder to know which action deserved the outcome. I need to distinguish variation between tasks, noise in the judge and numerical error in training. They affect different parts of the learning loop. None can be estimated from the number 612 alone.

Task variation and delayed credit

The RL reward curve mixes successive policies with different draws of problems. A higher point can reflect an easier batch, a better policy or both; the pool change makes that ambiguity larger. Multiple attempts on one problem are useful comparisons, but they are not the same as more independent problems. A defensible learning claim needs matched held-out evaluation and within-problem comparisons, with uncertainty grouped at the problem level.

There is a second issue even with a deterministic judge. A final score follows a long sequence of planning, edits, failed compilations, tests and summaries. Broadcasting that outcome through the trajectory supplies coarse credit: it does not identify whether a new move operator helped, a repair restored correctness, or a summary lost the best program. That is a reason to collect verified improvement transitions for the curriculum. It is an analysis of the objective, not a measured law that reward variance increases with episode length.

The same program can receive a different score

The takeover fidelity check re-graded 34 saved programs. Thirty-two reproduced their logged score exactly. One timed program scored 48.27 when logged, then 52.80 and 48.27 on two re-grades: a 4.53-point spread with unchanged code. The other exception was much smaller, 52.405 versus 52.369. The restored environment checks matched; runtime-sensitive judging still varied.

That example is large enough to matter when a proposed edit claims a small improvement. It is not an estimate of every task’s noise, or a claim that all scores have a ±4.53 error bar. Repeated submissions also create more chances to observe a lucky maximum. Comparing the largest score seen during two episodes can therefore reward extra judging as well as better code.

The takeover primary readout instead used the mean of three fresh grades of each selected program, with judging interleaved across arms, then paired the outcomes by saved state and clustered uncertainty by problem. This reduces dependence on a lucky in-episode score. Three grades do not remove all timing noise or uncertainty from adaptive program selection. For stage two, I want separately judged before-and-after programs and correctness checks before calling an edit an improvement.

Numerical gradient error is a separate measurement

A saved probe of the 9B outcome-RL path compared reduced-precision gradients with an fp32 reference on six real chains, 11K–171K tokens. The combined gradient had cosine similarity 0.82 with the reference; individual-chain similarities ranged from 0.57 to 0.88. This measures agreement of gradient directions. It is neither a reward-noise estimate nor an observed fraction of useful training examples.

The SFT audit leaves another explicit limit: its independent forward check included chains up to about 260K tokens, but it had no independent backward check in the 131K–260K band, containing 1,309 of 2,487 chains. That gap is a reason to validate long-sequence gradients. It is not evidence that those gradients were wrong. The six-chain RL probe also cannot certify every SFT chain or establish how numerical error changes with length.

These distinctions change the next experiment. More demonstrations can broaden capability coverage; repeated judging can make small gains more credible; precision checks can validate the update. None substitutes for the others. A shorter construction task makes each easier to diagnose before I add a long chain of feedback, edits and compactions.

My first working explanation is that the demonstrations were too narrow for the transfer I was asking for. Four attempts on each of 153 problems gave the student about 204 million loss tokens, but that is not 204 million independent lessons in algorithm design. The same representation, scoring function and failure mode can recur for thousands of tokens inside one episode.

What “a trace” means in this dataset

The distinctions matter. A median of 336K targets does not mean the model held a 336K-token context at once. Compaction split an episode into chains; writing a summary was itself a supervised call. Nor are the 336.75M forwarded tokens all new targets. Counting them that way would inflate the apparent amount of teaching by including reused context. The data was expensive and long, while its coverage was still concentrated in 153 tasks.

Analysis · More tokens and more experience are different interventions

That makes “612 traces were not enough” plausible, but not established. I have no data-size sweep with the recipe held fixed. More of these same trajectories could teach more of their long-thinking distribution, while more problems or more verified algorithm-building examples could teach something different. I need to count distinct problems, algorithm families, successful constructions and valid improvement transitions separately. The last two are coverage measurements still to make, not numbers I can infer from the token total.

Neither software-repair study demonstrates the data requirement for a 9B model learning unfamiliar optimisation algorithms over multiple compactions. They make the question more precise: how much relevant, learnable experience does this student need? A useful scale test would keep the objective and quality rules fixed, and distinguish more attempts on the same problems from genuinely broader problem coverage.

The second explanation concerns the student’s starting skill. My earlier blind reads found working search in roughly 3–4% of untrained 9B attempts on two held-out sets. So I should not call the capability literally absent. Those labels were also broader than the later strict sound-search check. The defensible claim is that working search was rare and unreliable under the tested sampling budgets, and usually lost to greedy code. Knowing an algorithm’s name, or finding a high-scoring greedy program among many samples, does not establish that the student can implement and maintain that algorithm.

Finite pass@k cannot prove that a model will never produce a skill. ProRL also reports broader pass@k coverage after prolonged RL, including on initially weak tasks. That is a reason to keep a curriculum as a testable route, rather than turn these short runs into a general boundary on what RL or a 9B model can learn.

The transfer I attempted bundled two capabilities. One is constructing an advanced algorithm: choosing a representation and moves, implementing a correct objective or score delta, and making a time-bounded search execute. The other is using that program as an agent: interpret feedback, make a useful change, check it, retain the best valid version and continue after compaction. My proposed curriculum teaches and measures them separately.

Stage 1 · One-shot algorithm construction

Ask for one complete program, with its reasoning, in a single response before any tool or judge feedback. That isolates construction from the later tool-using episode. Teacher demonstrations should cover different representations and algorithm families, with successful programs independently checked. SFT can introduce those constructions; OPD can then score the student’s own attempts; outcome RL can improve the resulting policy. That training sequence is a proposal for acquiring the algorithm, not a guarantee that the final stage will retain it.

The check needs to read and execute the program. It should distinguish a real search from greedy code with a timer, test a claimed incremental score against full recomputation when applicable, and check feasibility, termination and performance against a simple baseline. On unseen problems I would report both pass@1 and pass@k under a frozen definition of a working algorithm, plus score. Here a pass means a verified algorithmic construction under that definition, rather than a perfect benchmark score. A rare success at large k would be a foothold to strengthen; it would not yet be a reliable first attempt.

Stage 2 · Feedback-driven improvement with correctness intact

Start training continuations from verified programs, including the student’s own, and teach the edit–test–submit loop. The teacher’s useful lesson is the transition: what observation justified the edit, which invariant could it break, how was it checked, and did the re-graded program actually improve? Preserve a best valid version and evaluate correct-to-incorrect changes separately from score gains. More submissions alone would not pass this check.

This stage needs experience in the student’s own states. SCoRe finds that improving a first answer and learning to correct it are different objectives: single-turn training can improve the first attempt while worsening the second. Its method and short math/code setting differ from this proposed curriculum, but the distinction is directly relevant. It motivates training improvement explicitly, rather than assuming a competent one-shot solver will acquire it automatically.

Gandhi et al. provide another reason to teach prerequisites: priming a 3B model with verification and backtracking behaviours improved its subsequent RL learning on Countdown. That supports testing a staged acquisition of skills; it does not establish that this particular search-and-agent curriculum will work.

Integration · Restore the long horizon

Once construction and improvement each work, require them together from the problem statement alone. Extend the episode until it crosses compactions, checking that the summary retains the algorithm’s representation, invariants, outstanding failures and best verified program. Ultimately the student must recreate and improve a search without being handed teacher code. That is the full task the 612-trace experiment attempted from the start.

This also makes the diagnosis testable. Better one-shot algorithm pass@k without continuation gains would locate the remaining gap in improvement or memory. Gains only from teacher-supplied programs would leave independent construction unresolved. Success on short episodes that disappears after compaction would identify a horizon-dependent failure to investigate. Reporting those separately is more informative than one blended final score.

The already planned next run fits the first stage: teacher demonstrations, then OPD, then outcome RL in single-call mode. It starts from the released 9B, uses a clean 148-problem training pool, and gives teacher and student the same response budget: 96K thinking plus 32K for code, 128K total. Reference evaluations must be re-anchored at that budget. The trained response includes thinking, and selected demonstrations must finish with a compiling, non-degenerate program that beats the problem’s baseline.

A larger dataset and a curriculum answer different questions. More varied demonstrations may supply the missing constructions. Training improvement separately may teach the student to keep those constructions useful after feedback and edits. I want to measure both, before asking the student to imitate another full 90-minute expert episode across multiple compactions.