The Wall, the Compile Tax, and the Oracle

Part 1 ended with single-call GRPO taking Qwen3.5-9B from 6.22 to 12.22 Avg@5 on FrontierCS by step 70, with most of the gain by step 40. I then restarted from step 40 twice, each time with a recipe aimed at the obvious suspect, lost exploration. One restart used Clip-Cov, which stops the gradient on the tokens most responsible for entropy collapse. The other rewarded the best answer in each group: 75% of the advantage came from a leave-one-out best-of-4 score, 25% from ordinary GRPO.

Two things rule out "just train longer". Training reward rose from 0.10 to 0.26 by step 40 and then sat at 0.30–0.35 from step 50 in all three runs. And the model stopped improving on the training problems themselves. Each problem came back about every 14 steps, and reward on the same problem climbed visit by visit until the fourth visit, then went flat (Figure 2). The best of 8 samples on a problem stayed stuck at 0.46 from that point.

It isn't a lack of learning signal either. After step 20, 85–95% of groups still had a spread of scores, and all-zero groups were under 5%. Entropy held at 0.65–0.69. A 100-step run cost about 87 GPU-hours, so compute is small, and large-scale RL studies find that bigger batches raise the ceiling; 64 rollouts per step is small. My working conclusion is that the 9B model hit a wall of model plus recipe, and more steps of the same recipe would add at most about a point. This corrects part 1, which read the plateau as overfitting the 200-problem pool while training reward kept rising; in fact training reward went flat too.

Splitting the gain from the base model to Clip-Cov's step 70 into "more valid answers" and "better valid answers" shows what RL actually bought. A valid answer here is one that compiles and finishes within the budget (part 1's 43% → 86% counted compiling only). Valid answers rose from 38% to 83–85%, and the number of problems (of 172) where all five samples got the same nonzero score went from 2 to 10. The spread across a problem's valid samples halved.

A 172-problem eval with 5 samples per problem resolves differences of about 1.5–2 points, and every difference among checkpoints from step 50 on is 1.2 or less, none significant. Training reward and eval Avg@5 correlate at 0.90 across all checkpoints, but only at 0.12 after step 50. The eval can see the climb, but not the ranking at the top.

There is also a train/eval mismatch worth knowing: 74 of the 172 eval problems are interactive, and none of the 200 training problems are. Interactive problems score 7.2 on average against 16.2 for the rest.

FrontierCS problems are optimisation problems with partial credit, so the good solutions are often search: simulated annealing, randomised restarts, timed local search. Comparing programs within the same group, on the same problem, shows what the reward was teaching:

• Search-style programs fail to compile 16 points more often than safe ones.
• Once they compile and are judged, they score 0 about as often as safe code (−0.5 points).
• When they score, they score +4.3 higher.
• There's no time-limit signature: the risk is getting the program to build, not running out of time.

A mean-centred reward sees a group where the safe programs reliably earn a modest score and the ambitious ones sometimes earn more but often earn nothing. Its gradient pushes toward safe.

Reading paired base and trained solutions in full (10 problems, base vs Clip-Cov step 70) showed a pattern the numbers only hint at. The trained model often states the better algorithm, or names the flaw in its own approach, and then submits the safer program anyway. In its own words: "simple is better", and "It's safer for TLE." On one problem the score went from 99.9 for the base model to 48. On a maximum-independent-set problem it went from 86 to 34, and on a sphere-spreading problem from about 99 to 59.5.

RL didn't fail to teach the model good algorithms. It taught the model which of its own algorithms is safe to submit under a mean-reward objective.

If the problem is that the policy picks safe over strong, the natural fix is to reward the best answer in each group instead of the mean. This is the max@k family of objectives: optimise the best of k samples rather than the average sample. It avoided Clip-Cov's loss of search-style code (13.4% vs 7.8%, +5.6 points [+3.3, +8.0]), matching plain GRPO. Best@5 was 20.0 vs 18.6–18.8 for the other runs at step 70, but that difference is not significant. Quality on valid answers at step 50 was +1.36 [+0.04, +2.72] over Clip-Cov, the only significant score gain between the restarts. But Avg@5 slipped from 12.7 at step 50 to 11.8 at step 100 (not significant), and the run's stop rule ended it.

In theory and in a toy model, the reason is structural rather than a tuning problem:

• Copies of the best answer are never penalised; they just stop being paid. Each extra copy gets much less credit (Figure 6), and once five or more of eight samples tie for the best, the best-of-group term is exactly zero for the whole group.
• In converging groups, the remaining GRPO term takes over: 88–100% of the update, where per-group standard-deviation scaling turns a ±0.002 score gap into a ±0.43 advantage.
• All-tied groups carry no gradient and are dropped by dynamic sampling (which keeps only groups whose samples differ), so the share of converged problems tends to ratchet up.
• With deterministic scores, collapse is the optimum. The objective is maximised by always emitting the single best program; diversity only pays as a hedge, and the hedge costs mean score. In a toy model with a 75% best-of-4 weight, the policy reaches 0.969 best-of-4 but a 0.755 mean, below the 0.800 of always playing the safe heuristic. The real run's Avg@5 slipped (not significantly) while Best@5 held, which is consistent with this but doesn't prove it.

The agentic arm now runs in a minimal loop of my own: the problem statement in the first message, a handful of tools, and a submit that runs the benchmark's official feedback, up to 100 judged submissions per trial in training (unlimited in eval), scored as the higher of the best judged submission and the final file. The training reward discounts improvements by submission index (0.98 per extra submission, floored at 0.25). Trained from the base model with GRPO (and, from step 18, a 50% weight on the best of the group), per-step training reward was noisy (0.35–0.49) with no clear trend; reward on revisited problems rose about 0.02–0.04 per revisit. The eval didn't move.

The three base scores differ because the harness was revised between evals; each row compares checkpoints on the same revision. The first revision had an early-ending bug that lowered both arms by about 1.4 points.

The traces at step 20 explain the flat eval. Even untrained, the agent already used the judge as its test suite: it rewrote the file, submitted, read the score, and repeated. Only about 15% of judged submissions follow a local run, and training didn't change that (14.6% → 14.8%). RL added about 12 submissions per trial (median 38 → 50), all after the last improvement, mostly a habit of resubmitting to finish: endings at the repeat limit went from 9 to 20, and duplicate submissions stayed flat at about 25%. The four attempts on a problem became more alike (all four identical on 23 of 39 problems, up from 18); the worst attempt improved (+1.15, significant) while the best didn't (−0.69), and thinking per call fell 8%. With up to 100 exact-score submissions, the judge was nearly free, so RL never had a reason to make the model test itself.

The single-call model's gain also didn't carry over: run in agent mode, step 40 is +0.53 [−0.70, +2.23] over the base model. The two modes don't share as much as I'd hoped.

Short thinking is the model's habit, not the prompt's

In part 1 I suspected the harness was suppressing thinking. A 2×2 prompt ablation says it's mostly the model: a neutral task prompt and a "think first" system prompt changed thinking per call by −38 [−111, +33] and −26 [−95, +49] tokens from a mean of 566 over the first 20 calls (part 1 quoted medians, which are lower). The planning call runs about 1.3K tokens in every arm, while the same problem asked as one question gets 42K or more, 15–45× more. Removing "submit early and iterate" from the prompt changed nothing either (9.0 vs 9.1 submits per 10 minutes).

Reward per test case (from step 40, 16 samples × 8 problems)

Both arms score each test case separately, drop the per-group standard-deviation division in favour of one batch-wide scale, and give −1 to invalid code. That last choice risks making the compile tax worse, and it switches to 0 if search-style code keeps falling.

• Arm R, rank-based (after RiVER). On each test case the winner gets +1, the rest a bounded rank value, and a failed case −1. RiVER reports that a reward shaped like this, emphasising top-ranked solutions, improved rating on heuristic contests and transferred to exact-solution benchmarks, where raw-score training didn't. My change: tied winners share the bonus, so copying the best is paid less.
• Arm D, fixed quality bars. Each case pays 1.0 at or above a frozen top-10% bar and 0.5 at or above a top-30% bar, set from historical training results, never from the eval. A merely valid program earns 0, against −1 for invalid code. This loosely follows recent work on RL for code optimisation, which composes correctness with a speed objective instead of rewarding correctness alone.

What would count: the in-sample per-visit curve rising past the fourth visit, search-style share and Best@5 holding, fewer all-identical groups, and an Avg@5 gain over the step-70 checkpoints of all three earlier runs that the eval can resolve.

A scarce oracle (agent mode, from the base model)

OpenAI's o1-ioi used hand-engineered inference strategies designed for the 2024 olympiad; o3 reached gold without hand-crafted strategies. The search moved inside the model. My agent never had to learn it, because judging was nearly free. So the training run now allows 10 judged submissions per trial (with a visible counter), and feedback is only the verdict and a 10-point score band, never the exact score. The reward still uses the exact hidden score, and the eval stays in the benchmark's official mode.

What would count: the model starting to check its own work. That means more local compile-and-run before each judged submission (the share of judged submissions whose exact code was run locally first), brute-force checkers and stress tests appearing in the traces, those behaviours predicting higher scores, and the gain showing up in the official eval. The risk is that a 9B model barely self-tests today: about 15% of judged submissions follow a local run, and a brute-force checker plus generator appears in under 1% of trials. RL mostly amplifies what's already there.

The series is about collective intelligence: whether a team of copies of one model can learn to do more than the same model sampled more times. I don't think it's test-time compute in a different costume, but I'll only find out with a recipe that produces a different scaling curve. People specialised once communities grew past a certain size; the hope is something similar once a team does.

This post is a warning for that plan. Mean-centred RL narrowed diversity in the single-call runs, and rewarding the best answer only partly held it. A team reward based on the best answer would make copying a teammate optimal, by the same argument as Figure 6. So the team experiment is planned to:

• credit each agent by its marginal contribution, not by the team's outcome alone;
• give duplicate solutions, detected from per-test score vectors, only a fraction of the credit;
• sweep team size from 1 to 6 at matched tokens, against N independent agents with a best-of-N pick and one agent given N× the tokens.

If teams are more than extra samples, the online team's curve should bend above best-of-N past some team size, with measures of specialisation rising at the same point. I haven't found published work showing roles emerging among shared-weight agents after RL, and I'm not assuming they will. A flat solo curve at least makes any team effect easy to see.

RL learns what you pay for. Pay for valid code, and you get safe code. Make the judge free, and it becomes the test suite. The next experiments change what is paid for.