Climbing From Zero Without a Teacher

Today the usual way to give a small model a skill is to distill it from a bigger one. But someone had to go first, with no stronger model to copy. AlphaCode (2022) sampled up to about a million programs per problem, filtered them on the example tests, and placed around the middle of human contestants. o1-ioi added reinforcement learning and hand-engineered test-time strategies: 49th percentile at IOI 2024 under contest rules, and just above the gold cutoff with 10,000 submissions per problem. An early o3 checkpoint, evaluated on the same contest afterwards, scored gold under the 50-submission limit with no hand-made strategy; the authors credit scaling general-purpose RL. DeepSeek-R1-Zero showed that RL applied directly to a base model can grow long reasoning. None of them had a stronger model to copy, though each started from a large pretrained model, and AlphaCode was also fine-tuned on human contest solutions.

That's the situation I care about. If the model you're training is the strongest one you have, there is no teacher. You have to hill-climb: sample, keep what scores better, repeat. RL can only reinforce what the model actually samples. In GRPO, a group of attempts that all score the same carries no signal at all, which is why recipes like DAPO filter those groups out. So climbing from almost zero needs two things:

• a foothold: the model must sometimes produce the skill (STaR names the same limit: its bootstrapping needs a starting model that already does better than chance), and
• a slope: each step toward the skill must score higher than the step before.

The skill here is search: simulated annealing, local search, beam search, the programs that win FrontierCS's open-ended optimization problems. Parts 1 and 2 raised the 9B's score. This part asks whether any of that was climbing toward search, and what happened when I tried to build the hill directly.

FrontierCS problems give partial credit, and a surprising amount of it goes to programs that barely try: print a trivial valid answer, or run one greedy pass. Part 2's gains in Avg@5 were real, but they didn't say what kind of program earned them.

So I built an algorithm score: for each problem, it credits only the points more than 1 above the best trivial program for that problem. A model that just learns to ship valid code more often barely moves it; a greedy program can earn credit, but only by beating the trivial bar.

If your metric can't tell "learned a better algorithm" from "learned to ship valid greedy code", you'll celebrate the wrong thing.

A working search for these problems needs three things: a loop bounded by a timer, an incremental way to score a small change, and keeping the best answer found so far. The 9B talks about all three. It writes them almost never.

• Handed a search family in its opening thought or in the prompt (untrained 9B, 10 judged submissions), it attempts search more often, up to 28% of programs vs 7% for controls, but writes that family at most about 13% of the time. Time guards appeared in 3 of the first 308 programs (automatic labels).
• Started from its own archived programs (mostly search) and paid only for beating them, the step-40 checkpoint produced 0 working searches in 127 attempts, against 0 in 128 from scratch (a strict static check; the blind-read rate below is a looser label on the untrained model). Here a working search is a static check: a search class, a timer inside a loop, and a clean run. It patched its own scaffolding or rewrote it as greedy, and never added a timer-guarded loop.

So there is a foothold, barely: in a blind read, 3–4% of the untrained 9B's attempts on held-out problems are a working search (§6). The problem is the slope.

A loop with no clock runs out of time. A clock with no cheap way to score a change makes too few moves to beat the greedy start. I didn't score each half-built stage separately, but search attempts as a whole score below the greedy program they were meant to improve, and even the valid ones only draw level with it. So the expected reward favours greedy, and RL has no reason to move toward search; in these runs the search share stayed flat at 3–19%. Part 2 called this a compile tax. In agent mode it's an execution tax.

Each lever borrows from a line of prior work that climbed from a weak start, and each tries to fix one thing: the slope (reshape the reward), the environment (make checking pay), the foothold (hints and scaffolds), or the sample pool (learn from the model's own best).

Most had a check written down before they ran, and several had a kill rule: a stop condition that ends the run early if the first signal isn't there. They start from either the untrained 9B or the step-40 RL checkpoint, and are read on different instruments, noted in each write-up. Avg@5 is the mean over 5 samples per problem; Best@5 the mean of each problem's best.

per test case, rank the group's 16 samples
  1st          +1
  2nd          +0.5
  …            falls with rank
  15th–16th    −0.5
  fails case   −1
reward = mean over test cases
invalid program = −1

per test case, against bars frozen
from 17,316 past training samples
  ≥ top-10% bar   1.0
  ≥ top-30% bar   0.5
  below           0
invalid program = −1

reward = score + 0.3 if all three:
  ✓ its own code calls the kit's driver
  ✓ valid on every test
  ✓ full budget beats a run with
    the search skipped by > 1.1
a timer in a loop doesn't count

$ submit solution.cpp
  verdict   accepted, partial
  score     in [70, 80)
  left      3 / 10
exact score and per-test results hidden;
the reward still uses the exact score

<think>
I'll use simulated annealing: score each
move by its delta, cool over time, stop
before the time limit. The solution state,
the move types, and how to compute each
move's score delta in O(1) are:
▌ the model finishes the sentence
~68 forced tokens, masked from the loss;
reward only counts points above the trivial bar

#include "kit.h"   // 72 lines, training only
  deadline = 0.85 × time limit
  drive(propose, delta, apply, undo)
  anneal(...)  best_so_far  rng  fast_io
prompt: start simple and valid, then
improve with a time-bounded search

21,940 training rollouts → keep a sample if it
  scores wherever any rollout scored
  is within 0.02 of the problem's best
  beats trivial, sits above the median
165 examples on 70 problems
→ full fine-tune, 4 epochs

start from one of its own archived programs:
  [its own archived program]
  + its verdict, score and slowest test
task: repair · improve · upgrade it
paid only above max(bar, parent) + 0.02
8 parent groups + 8 from-scratch groups

Each lever in detail, in the same order: the setup, what I wrote down before the run, what happened, and my best reading of why.

Rank reward per test case

Fix the slope · after RiVER · from step 40, single-call · 16 samples × 8 problems per step · steps 40–60

Setup

Each test case ranks the group's valid samples. A unique winner gets +1, the rest a value that falls with rank from +0.5 to −0.5, and a sample that fails the case gets −1 for it. Tied winners share the average of their ranks' values, so copying the best pays less. A sample's reward is its mean over cases, and invalid code scores −1. Advantages use one batch-wide scale instead of each group's standard deviation.

Written first

Part 2 (§7) listed what would count: the in-sample curve still rising, search-style share and Best@5 holding, and a paired Avg@5 gain. The stop rule: entropy below 0.3, gradient norm above 3× its trailing median, validity under 70% for 5 steps, or two evals in a row without a paired gain.

What happened

Validity rose from 72% to 81%, but at step 50 on the 172-problem set Best@5 fell 3.19* against step 40 (Avg@5 −0.78 [−2.20, +0.53]), and it trailed the other runs' step-50 checkpoints too. The algorithm score went from 0.27 to 0.17 (n.s.). A blind read of 40 valid programs from steps 46–52 across five runs found 26 re-implementing the baseline described in the problem (6 of 8 in this arm) and 1 doing something non-trivial. This was graded while another job loaded the grader's CPUs; the clean regrade never ran. Stopped at step 60.

Why

Rank credit pays for being best in the group, and the group was mostly greedy code and baseline copies, so "best" meant the best greedy program. Search code was invalid about a third of the time, and at −1 per invalid sample it netted below greedy (§3). The reward bought validity, not search, and the top end got worse.

Fixed quality bars

Fix the slope · from step 40, single-call · same batch and advantage as the rank arm

Setup

Each test case pays 1.0 at or above a frozen top-10% bar, 0.5 at or above a top-30% bar, and 0 otherwise; invalid code scores −1. The bars came from 17,316 judged training samples of three earlier runs, never from eval problems.

Written first

The same criteria and stop rule as the rank arm (part 2, §7).

What happened

On training samples (steps 41–52), validity rose 3.0 points [+0.3, +5.7]* and the non-trivial share didn't move (+0.4 [−1.1, +1.9]). The step-50 eval was stopped at 490 of 860 grades to free compute for the strategy probe and the algorithm score, so there is no eval result.

Why

The bars barely separated anything. Scores tie so often that half the historical samples clear the "top-10%" bar, and the two bars coincide on 69% of test cases. A copy of the problem's baseline clears the top-30% bar on 76 of 165 problems and earns the full 1.0 on 27. Pooled over this arm's own samples, exact baseline outputs earned more reward (0.59) than above-baseline ones (0.48). And at −1 for invalid code, search netted −0.14 against +0.01 for greedy.

A kit bonus

Fix the slope · from step 40, agent mode · 25-problem pool · kit and strong prompt offered to 6 of 8 attempts per group

Setup

The reward is the raw score plus 0.3 when three checks pass: the program's own code calls the kit's search driver with real move functions (a timer in a loop isn't enough); it is valid on every test; and it scores more than 1.1 points higher at full time budget than in a re-run with the driver's search skipped.

Written first

A review pause before step 11, with one change allowed in advance: raising the bonus. No kill rule.

What happened

Over five steps, 1 driver call in 480 offered attempts and no bonus paid. Offered attempts averaged 0.317 reward against 0.332 for 160 controls, and time guards fell from 27 to 14 per 96 kit attempts over steps 1–3.

Why

RL can only strengthen behaviour that gets rewarded, and a bonus nobody earns sends no signal. Defining the bonus was harder than expected. An earlier design paid when the full-budget score beat a quarter-budget run; checked offline on 315 recorded kit programs, it missed 29 of 45 real searches (real searches converge early) and sent 31% of its payouts to greedy constructions that a timer happened to cut short. The stricter driver-only gate fixed that and then paid nothing.

Scarce oracle

Make checking pay · from the untrained 9B, agent mode · 8 attempts per group · 60-minute trials · steps 1–31

Setup

In training only, each trial gets 10 judged submissions with a visible counter. Feedback is a verdict and a 10-point band, such as "score in [70, 80)", never the exact score or per-test results. Identical resubmissions are free. The reward still uses the exact hidden score, and the eval keeps the benchmark's normal unlimited, exact feedback.

Written first

Part 2 (§7): the model starting to check its own work. That meant more local runs before judged submissions, brute-force checkers and stress tests appearing, those behaviours predicting higher scores, and the gain showing up in the official eval. There was no outcome kill rule; I stopped the run after step 31.

What happened

The first criterion was met (Figure 6): local runs per submission rose from about 2 to 3, and trials with a self-written helper script from 22–36% to 39–43%. The rest wasn't. The search-style share stayed at 3–19% with no trend, the step-10 eval was −1.01 [−2.26, +0.14], and a blind read found the same algorithms. The habit didn't carry into the uncapped eval (35.9 vs 36.9 judged submissions per trial).

Why

Checking only pays when there's something worth checking. When a search timed out, the model deleted the search; it never added a time guard. One caveat: the sampler kept the first 16 of 24 groups to finish and dropped 224 of 696 as slow. The dropped trials scored about the same (0.36 vs 0.40) with the same behaviour markers, so this tilts training toward easy problems rather than demonstrably against search, but the run isn't a clean test of scarcity.

Plan first

Give a foothold · after STaR-style hints · from the untrained 9B, agent mode · 16 groups × 8 trials per step · 10 judged submissions

Setup

On its first turn, after reading the problem, about 68 tokens are forced into the model's own thinking. They name a search family (annealing, local search, beam or exact/DP, drawn at random), give generic how-to (delta evaluation, cooling, a time guard), and end with an open stem the model must finish: "the solution state, the move types, and how to compute each move's score delta in O(1) are:". The forced tokens are masked from the loss, and the guided share fades from half the groups to none by step 20. The reward credits only points above each problem's trivial bar.

Written first

Five gates at step 20, read on 16 unguided trials per held-out problem against the base model. They covered the share of better-class programs, pass@16 and Best@16 not falling, entropy, held-out likelihood, and plan-to-code match of at least 0.70.

What happened

At step 10 the 40-problem agent eval was −1.05 [−2.33, −0.03]*. By step 20 the held-out score (set A) was up 1.8 [+0.1, +3.7]*, but the run failed two gates. The entropy ratio was 0.874, and the plan-to-code match fell from 0.59 to 0.45 as stating a plan fell from 56% to 44%. The better-class share passed narrowly, +10.9 points [−0.1, +21.4]. Mid-run I changed three things: guidance went only where unguided trials failed, the problem pool was re-picked for headroom, and a contest-style penalty for failed submissions was added. Read the step-20 numbers with that in mind.

Why

The reward can't see execution. None of 318 credited search programs was sound, only 4 had a time guard, and 65% of the positive push on search went to code that timed out or was wrong. The planning turn carries about 1% of the gradient weight. In training, 91% of positive advantage on guided trials went to code that didn't follow the forced family, though that match comes from an automatic classifier that agrees only weakly with blind readers (κ 0.26).

A search kit

Give a foothold · inside the plan-first run, steps 23–48 · offered to 6 of 8 attempts per group · training only

Setup

A 72-line header with a timer set to 85% of the time limit, a best-so-far holder, a random generator, fast I/O, and generic search drivers: propose, score the change, apply or undo, with annealing-style acceptance. It names no algorithm, and the eval never sees it. Over six phases it was offered as a workspace file, then as an include, then with a strong prompt ("start simple and valid, then improve with a time-bounded search"), then with a neutral prompt, then with the strong prompt held, then turned off.

Written first

Revert if offered attempts trail controls on clearing the bar by more than 5 points for 2 steps.

What happened

As a file it never reached final code: the model rewrote the whole file dozens of times per trial and dropped the 72 lines each time. As an include it appeared in 20–33% of judged code, but the search driver was never called in 413 submissions. With the strong prompt the kit appeared in 72–82% of code, with time guards in 18–24% and working search in 3–5% (Figure 7). Back on a neutral prompt the guards fell to 5–11%. Holding the strong prompt gave −9, −4, −9 and −13 points against controls, the rule fired, and the kit was off from step 47.

Why

The reward pays a bar-clearing greedy program as much as a bar-clearing search, so RL had no reason to keep the kit. The best kit program a blind reader found was a naive annealing loop that rescored the whole solution on every move. Attempts never offered the kit never wrote a time guard. "Working search" here is an automatic label (search class, time guard, above the bar, no timeout or crash) and carries the same κ 0.26 caveat. My own process was weak in places too: the fade ran on a timer rather than on evidence, and some decisions rested on one- or two-step windows with about ±10 points of noise.

Self-distillation

Amplify its own best · after ReST-EM · from step 40, single-call · 165 examples on 70 training problems

Setup

From 21,940 training rollouts of three single-call runs, I kept samples that scored on every test case any rollout scored on, came within 0.02 of the problem's best, beat the trivial programs, sat above the problem's median, and ended their thinking naturally. That left at most 4 per problem: 165 examples on 70 problems, about a quarter of them search. Full fine-tune for 4 epochs.

Checks

A memorisation gate and a likelihood-ranking probe.

What happened

At epoch 2 on the 172-problem set, against step 40 regraded on the same judge: validity 72.0% → 80.4% (+8.4 [+4.9, +11.9]*), truncated outputs 16% → 7%, Avg@5 +0.83 (n.s.), Best@5 −0.72, algorithm score +0.06 (n.s.). A blind read of 14 problems found the same algorithm on 7, epoch 2 better on 4 and step 40 better on 5. Epochs 3–4 failed the memorisation gate, and the ranking probe was null throughout.

Why

Training on its own best samples sharpened the model (entropy −9%) without lifting its ceiling: three-quarters of those samples weren't search to begin with. A caveat: this ran in a trainer setup later found to compute wrong gradients (cosine 0.80–0.88 against a reference), and I haven't rerun it, so treat it as confounded.

Archive-seeded expert iteration

Amplify its own best · after ExIt and FunSearch-style archives · from step 40, agent mode · 8 parent + 8 scratch groups per step

Setup

An archive of about 90,000 judged programs from earlier runs, classified statically. Of 33,295 distinct programs on the 30-problem pool, 25,155 were greedy; only 42 archived programs passed the static check for working search. Each parent group starts from an archived program (186 search parents to repair or improve, 21 greedy ones to upgrade), shown as the starting solution with its verdict, score and slowest test. It is paid only for beating both the bar and the parent's verified score by 0.02. Scratch groups start from nothing.

Written first

Stop if parent groups' working-search yield is below 2× the scratch groups', or if fewer than 7% of repair and upgrade attempts reach a verified above-bar program in the parent's class. The code also required a non-zero yield, which the written rule didn't cover; I wrote down the 0-vs-0 case after step 1, before step 2 was graded.

What happened

0 working searches in 127 parent attempts, against 0 in 128 from scratch. Conversion was 7 of 103 (6.8%). Upgrades produced 0 search wins in 40 tries; the 5 that were paid all got there with better greedy code. The rule fired at step 2, saving about 8 steps, roughly 150 GPU-hours.

Why

In 112 diffs the model either patched its own search scaffolding or rewrote it as greedy (18 local searches and 7 annealers became greedy). It never added a timer-guarded loop. Even handed its own search skeleton, it didn't know how to run it, and greedy was what got paid. Caveat: the static classifier mislabelled at least one "search" parent that was really the problem's baseline.

The scarce oracle is the closest I got to o3's story. The o3 report describes it often writing a simple brute-force solution and cross-checking its optimized code against it, a strategy the authors say emerged from end-to-end RL. Here, scarcity alone produced a weaker relative of it before any training: running code locally before submitting. I didn't measure whether those runs used a brute-force reference. The difference is what gets checked. A checker protects an algorithm that already exists; here it protected a greedy one.

Hints and scaffolds are how STaR and reverse curricula get a model unstuck: show it part of the answer, then take the help away. Here the help changed what the model wrote while it was there, and the habit left with it. Both the plan and the kit were prompts and a header, not training on examples of the skill; that difference comes back in §6.

Amplifying the model's own best output tells the same story. Expert iteration works when the expert reliably beats the apprentice: in ExIt the expert is a tree search, and ReST-EM keeps only samples that pass a correctness check. Here the natural expert is the agent, with its tools and judge, and it wasn't much of an expert. Comparing the agent's best program with the single-call best on 30 problems, the agent's was a genuinely better algorithm in only 6; in 12 it was the same algorithm, tuned. 81% of that agent-mode pool came from tuning against judge feedback. Distilling those samples would teach tuning, not search.

That reframes the problem. The 9B isn't missing an incentive that a clever reward could supply; it's missing a skill its bigger sibling already has (a different release in the same family, so size isn't the only difference). There, the foothold is wide and the slope points the right way.

That fits what others have found. In the DeepSeek-R1 report, distilling R1 into a 32B model beat more than 10,000 steps of RL on the same 32B base: 72.6 vs 47.0 on AIME 2024, and 57.2 vs 40.2 on LiveCodeBench. Yue et al. find that RL with verifiable rewards mostly samples the base model's existing successes more efficiently: at large k the base model catches up and passes it, while distillation adds new reasoning patterns. Gandhi et al. saw the same split at 3B: under identical RL, a model that lacked verification and backtracking plateaued, and priming it with examples of those behaviours, even ones with wrong answers, fixed it. The counterpoint is ProRL: with more than 2,000 RL steps, gains were largest on tasks where the starting model's pass@128 was lowest. That model started from a distilled checkpoint, and my runs were tens of steps, so I can't rule out that much longer RL would climb.

So what did the first climbers have that this run didn't? Much bigger models, far more compute (AlphaCode's million samples, the o-series' RL), and, for AlphaCode, human-written solutions to fine-tune on. In practice, "from zero" may have meant "from rare, in a big model": a foothold and a slope like the ones the 27B already has.

The goal I set for this direction was a 0→1 breakthrough in a world where the 9B is the strongest model available. Without a teacher, that jump didn't happen at 9B with any of eight levers. The next step tests the teacher route, and then the closest thing to a teacher-free foothold: code written by people.

• "It knows better than it ships" (part 2, §4). The model does name better algorithms and ship safe ones. But the gap between the best of 55 samples (29) and a single sample (~12) isn't evidence of a hidden algorithm: algorithmic credit is about 2% of the score at every checkpoint, and about three-quarters of the best training samples I distilled weren't search.
• "Now running" (part 2, §7). Both experiments have stopped. The rank reward made the top end worse, and the bar reward has no result. The scarce oracle raised self-testing but not algorithms or scores.
• "Agent mode" (part 2, §6). Step 30 confirms the flat eval: −0.52 [−1.18, +0.08], with mean judged submissions per trial rising from 47 to 77 (part 2 quoted medians).
• Comparing single-call and agent-mode scores. The 40-problem agent set was hand-picked to be hard, about 3.4× harder than the 172-problem set by one measure: single-call step 40 scores 3.34 on it, against 11.48. On the same 40 problems, the untrained agent (6.77) scores about twice the RL-trained single-call model.
• "The compile tax" (part 2, §3) got harsher under −1 for invalid code, and in agent mode it's an execution tax (§3).

1. On-policy distillation. The 9B samples, the 27B scores the exact tokens it sampled, and the per-token difference is the training signal, with no outcome reward (in the spirit of GKD). Training is stable so far, with no dropped samples; the divergence from the teacher was 0.089 at step 5 against 0.093 at step 1, too early to call a trend.
2. Seeding from human search code. Collecting permissively licensed human-written search programs as examples of the missing skill. This is the closest thing to a teacher-free foothold: the examples come from people, not a stronger model.
3. Longer horizons for the 27B. Without compaction, 11 of 20 27B attempts hit the context limit before a judged submission. A new compaction keeps a summary, recent turns verbatim, and the judge's record.
4. Teams. The team runtime is built on my own harness. Untrained, the 9B only forks when hinted. The collective-intelligence question comes back once a model has a skill worth dividing up.

To climb from zero you need a hill: a foothold the model sometimes samples, and a slope where each step toward the skill scores higher. At 9B, neither rewards nor scaffolds built one. With a foothold but no slope, the next step borrows the skill itself, from a model or from people, so training starts on the far side of the valley.