The Harness Is Part of the Model

Before training anything, I wanted to know what an off-the-shelf model does with teammates. I ran Qwen3.8-27B inside the Qwen Code agent harness on a 96-task eval set in six arms. Three are the harness's own modes: solo, built-in subagents (a lead delegates pieces to helpers with fresh contexts), and agent-team mode (a leader plus up to three teammates and a shared task list). The other three are versions of a collective harness I built, in which any agent can fork a copy of itself and agents talk peer to peer (§7 describes it briefly). Every arm used the same model, sampling, 262K context window and compaction rule, the same caps (1 h for FrontierCS, NL2Repo and KernelBench; native step caps for the tool and enterprise tasks), and one attempt per task.

The three versions of my harness: v1 has a shared workspace, fork with full context, messages and a whiteboard. v1 + submit adds a team-wide submit that ends the task. v2 gives each agent its own git worktree (a fork copies its parent's tree), grades the submitting agent's tree, and stamps the time left on every tool result.

The eval set was chosen to be hard for a single agent. I picked tasks only from published per-task difficulty, never from my own runs: low but nonzero scores for comparable models, and often failed by frontier agents. I dropped anything that overlaps the training pools, for example ten FrontierCS problems that are copies of FrontierSmith training problems. None of them is in the 172-problem set used for the 9B results.

So the honest headline is a null: on these tasks, an untrained 27B model doesn't get better by being given teammates, in either the harness's built-in team modes or mine. Three behaviours behind the numbers are more informative than the means.

Collaboration only appears where the work splits

In the collective harness the model never forked on tool workflows, enterprise tasks or GPU kernels. On FrontierCS, where the deliverable is a single optimised program, it rarely forked (v1: 2 of 80 trials; v2: 4 of 40). On NL2Repo, which means building a multi-module package from a spec, it forked often (v1 + submit: 15 of 22 trials, 44 forks, up to six agents at once; v2: 9 of 22 trials, 27 forks). The work divides by module, so the model divides it.

The structure it built was always the same, and nobody told it to: in v1 and v1 + submit, all 47 forks came from the first agent, one level deep. That agent made itself the lead, wrote a brief for each worker into the workspace, forked workers in waves, ran the full test suite and sent failures back to whoever owned the module. Workers did talk to each other, warning about broken imports. But the lead also spent a lot of time idle, waiting on messages, and the payoff was small (v1 + submit: forking trials +0.03 over solo; v2: −0.04).

The clearest collaboration was on one hard FrontierCS problem (v1 + submit). The lead forked two helpers to build a brute-force checker. One saw that its assignment duplicated the other's, took the orphaned test harness instead and told both, and later reported two compile errors back to the lead. That trial scored 0.486 against 0.176 solo, and it didn't recur in v2.

The best collective number came from one agent and new stopping rules

The second version of the harness gave every agent its own git worktree and showed time remaining in every tool result. Its best number, 0.309 vs 0.251 solo on 3-hour FrontierCS (10 tasks better, 8 worse, not significant), came from trials where the first agent worked alone and used the time: its median submission moved from about 43 to about 75 minutes. At 1 hour the behaviour flipped the other way: submissions rose from 17 to 37 of 40 trials, and runs that hit the time cap fell from 16 to 2. v2 changed worktrees, submit semantics and the clock all at once, and I haven't ablated the clock alone. Even so, its stopping rules changed when agents stopped more than any collaboration primitive changed what they did.

The submit action was adopted immediately, as a "done" button. That cost points: two FrontierCS submissions at minutes 4 and 27 scored 0, and four trials submitted while workers were still running.

Long tasks are compaction tasks

Between 77% and 90% of 1-hour trials compacted their context at least once, in every arm, some up to five times, and the median peak prompt was about 225K tokens. At this length "solo vs team" is partly "one lossy memory vs several". That's the regime where a team should help, and one more reason a single-attempt null is a starting point, not a verdict.

The untrained model has the mechanics of teamwork (forking, briefs, peer fixes, integration testing) but not the judgment of when it pays. That judgment is what RL is supposed to teach.

A team-RL result would mean nothing if the RL stack couldn't improve a single model, so the first training run was solo and as simple as possible. I trained Qwen3.5-9B with single-turn GRPO on the 200 FrontierSmith training problems and evaluated it on all 172 algorithmic FrontierCS problems with the benchmark's own partial-credit judge, 5 samples per problem. Scores here are on a 0–100 scale; §1 reported the same judge on 0–1. Avg@5 is the mean over the 5 samples, and Best@5 the mean of each problem's best sample.

For reference, the published FrontierSmith recipe reports 1.80 → 10.62 for this model after 100 steps at a 16K-token budget. In that setting my base model scores 1.19; at my 64K budget it starts at 6.22. So the comparable number is the gain, not the endpoint.

A 16K budget starves GRPO

At the published 16K-token cap with my 8-group batches, 98% of responses were truncated and every step-1 group scored 0. When every sample in a group scores the same, the group-relative advantage is zero and there is nothing to learn. The paper still learned at 16K, from the handful of problems that do earn reward; I didn't want to rely on that. At 64K, step-1 reward was about 0.15, with signal in 6 of 8 groups.

Teaching it to stop thinking

Even at 64K, more than half of the responses ran out of budget mid-thought. Injecting the model's own early-exit sentence at a 32K thinking budget wasn't enough. It kept reasoning in the answer (a median of about 22K more tokens), and 40 of 64 step-1 samples wrote a second </think>. What worked was a 56K budget followed by the early-exit sentence plus an opening code fence, both masked from the loss, and an 8K answer cap. That brought it to 9 of 64. The new cap did not make the base model better (−0.50 [−1.54, +0.51]), so the later gain is training, consistent with the model learning to finish. In the base model, samples whose thinking closed naturally scored 11.1 on average against 1.75 for force-closed ones.

Keep loops and failures in the batch

About 14% of base-model responses loop, typically detectable only around 50K tokens in. Dropping them would hide looping from the gradient and bias the group baseline upward, so loops run to the budget and stay in the batch as failures. An early-stop detector slowed generation 20–35% to save about 3–4% of tokens, so I turned it off. Retries are only for infrastructure failures, and those are masked, never scored as 0. I also dropped an overlong-response penalty that would have dominated the signal: mean step-1 reward was −0.62 with the penalty, while task scores were around 0.1.

Why it flattened

Dynamic sampling draws 12–16 problems per step to fill 8 groups, so by step 70 each of the 200 training problems had been drawn about 5.5 times and trained on about 3. On repeat draws, training reward roughly doubled (0.136 → 0.303). Groups where all eight samples got the same partial-credit score rose from 1% to 10%, while all-zero groups fell from 24% to 5%; identical groups carry no gradient. Held-out gains stopped. (I first read this as fitting a small pool; part 2 corrects that: training reward and reward on the training problems went flat too, so it's a wall of model and recipe.) I stopped the run at step 77 and kept the step-40 and step-70 checkpoints.

One detail matters for anyone reaching for an entropy-preserving fix. With one optimizer update per step, the importance ratio is exactly 1. The PPO clip never fires (clip fraction 0.0 at every step), so clip-higher and KL-Cov are no-ops: KL-Cov produced bit-identical updates. Fixes that act on the gradient directly, such as Clip-Cov's token masking, or more problems, are the levers that are left.

Why the 40-problem subset can mislead

For quick checks I also kept a 40-problem subset: §1's hard FrontierCS set, on which the base model scores only 0.47. On it, step 20 looked no better than step 10 (−0.16 [−1.35, +1.06]), while on all 172 problems it was clearly better (+1.40 [+0.05, +2.87]). The subset has only 10–16 problems the model can partly solve, too few to resolve a trend. It's useful as a smoke test, not for deciding whether training is working.

A team has to run inside an agent harness, so the next step was the same model, problems and judge, trained as a solo agent in a coding harness (Qwen Code). That meant a 262K context window, many short model calls per trial, and deadlines of 15–30 minutes. The recipe was token-level GRPO with a standard-deviation floor on the advantage, KL 0.01 to the base model, one update per batch, and parallel dynamic sampling (launch 12 groups × 8 trials, keep the first 8 groups with a reward spread). Only the model's own sampled tokens were trained. It isn't identical to the single-call recipe: it uses a token-mean loss, a standard-deviation floor, KL 0.01, a wider clip, a warmup, and an 8K per-call thinking guard (which fired on only 0.03% of calls).

Training looked healthy: rewards 0.18–0.33, gradient norm about 0.15, no dropped trajectories. The eval disagreed.

At step 10 both arms' KL to the base model was about 0.0003; the single-call arm's then kept rising (0.0015 at step 20, 0.0026 at step 30), and the agentic arm's barely moved. Changing the KL coefficient would have been the reflexive fix, but at that drift the KL term's pull is about a thousand times weaker than the reward's, so it was the wrong lever. Reading trajectories found the problem. On the same packing puzzle:

• The single-call model at step 40 thought for about 30K tokens, worked out that the problem is NP-hard, designed a best-fit greedy over all eight rotations and reflections, wrote one program, and scored 16.9.
• The agentic model at step 10 made 178 calls and scored 0. Each call carried one sentence of narration as its "thinking". It rewrote its solution from scratch about ten times, once writing JavaScript into a .cpp file, called tools that don't exist, and "verified" an output that placed nothing.

Across agentic evals, thinking was a median of 27 tokens per call. 3.5% of eval trials ended with no solution file; in the one I traced, the agent deleted it during cleanup. There are three likely contributors:

• The problem arrived as a file to read rather than as the question, so the model treated it as a quick tool result.
• The roughly 13K-token system prompt is written for software engineering ("briefly state what you're about to do").
• The model's own habit of one sentence of reasoning per tool call. My minimal loop below still thinks a median of only 41 tokens per call.

The harness also gave the agent no local scorer, although the benchmark's own agent mode provides one.

I stopped the run and replaced the general-purpose harness with a minimal loop of my own:

• the statement goes in the first message;
• a handful of tools, plus a submit that runs the benchmark's feedback script;
• exact token ids captured on every call.

It follows the benchmark's official agent mode: unlimited submits in eval (training caps them at 100), credited with the best of the final and best submission, which is a different metric from the evals above. In a first smoke test, the base model in that loop still thinks briefly per call (median 41 tokens), but its planning call runs to a few thousand tokens, and it submits 16–39 times per trial against real feedback. Whether that trains is the next result.

A healthy training dashboard can sit on top of a policy that isn't learning anything useful. Read what the model actually does, and treat the harness as part of the model.

Agentic RL is expensive in a way single-call RL isn't. Per training step, the agentic arm used about 8× the GPU-hours of the single-call arm while generating about the same number of tokens. Most of this project's engineering went into closing that gap without touching what the model learns. Two rules made that safe:

• Push speed until the next lever is worth less than 5%, then launch. Don't optimise during a long run. The one exception here, the surplus-group early stop, went in mid-run only after a replay showed the kept set unchanged.
• An infrastructure-only change must leave the gradient unchanged, and that has to be shown by replay: the same kept groups, gradient cosine at the noise floor, byte-identical batches. Speed changes are also measured off the live run, because faster serving changes how many turns fit before a deadline.

Serving many long-context agents

The workload is unusual. Up to a dozen agents share each inference replica, each holding up to ~200K tokens of context and taking short turns between tool calls. The biggest lesson is that decode is memory-bandwidth-bound: what pays is keeping the right prefixes in cache and the right number of agents per GPU, not adding raw capacity.

Between rollout and trainer

The agentic arm's step was ~89 minutes at first. Token-in/token-out alone brought it to ~47. The other fixes below are smaller, two are built but not yet in a completed run, and every one was checked to leave the update unchanged.

Evaluation and grading

Grading is CPU work with time limits, which makes it fragile in a quiet way: overload doesn't crash, it lowers scores. The biggest fixes here were about trust, not speed.

None of these crashed a run. Each one either fed the optimizer a wrong signal or would have turned a real result into noise.

1 · Normalising by a tiny standard deviation amplifies noise

GRPO divides each group's reward differences by their standard deviation. With partial-credit rewards, a group of nearly identical scores, 0.549 / 0.549 / 0.550 / 0.553, becomes advantages of −0.74 / −0.74 / −0.22 / +1.69: a full-strength gradient on what is basically timing noise. Dropping the division (A = R − mean, which works for 0/1 rewards) went too far the other way: the gradient norm was 0.022, 10–20× too small for 0–1 partial credit. The fix is a floor: A = (R − mean) / max(std, 0.05). The agentic arm used it; the single-call arm, which learned, used plain normalisation. So this is about noisy partial-credit groups, not why the agentic arm stalled.

2 · Sequence-parallel gradients on a hybrid model

Qwen3.5 mixes full attention with Gated DeltaNet linear-attention layers. In my full training stack with Ulysses sequence parallelism, the forward pass matched plain data parallelism but the backward pass did not. On a 47K-token sequence, the gradient's cosine similarity with data-parallel training was 0.851 (0.787 in the linear-attention layers, 0.666 in the embeddings), against a run-to-run noise floor of 0.9999. A second check on a 67K-token rollout, through a different trainer code path, found the lm_head gradient off as well: cosine 0.682 at 1.39× the expected norm. A head-parallel implementation of the linear-attention layers brought the first check back to 0.99991. For the single-call arm I simply turned sequence parallelism off and offloaded optimizer state instead.

I haven't pinned down the cause. On a second machine with a different GPU architecture, I ran the linear-attention layers alone at 2 and 4 sequence-parallel ranks, on multi-chunk and packed sequences, with and without activation checkpointing. The parameter gradients matched a full-sequence reference to cosine 0.99998 or better, and a control that cut the cross-rank state moved the outputs at least 10× more than sequence parallelism did. So the problem is either in how the pieces are put together in the full stack or specific to the original hardware. Two lessons hold either way:

• Log-prob parity checks only test the forward pass. Before trusting any parallelism scheme on a new architecture, compare its gradient with plain data parallelism on one real sequence, in the exact stack you train with.
• Random initialisation hides this class of bug. With default initialisation these gates forget their state within a few tokens, so almost nothing crosses the rank boundary and a gradient test on random layers passes no matter what. Set the decay to trained-model-like values first.

3 · Optimizer state that appears on step 2

Adam's moment buffers don't exist until the first optimizer.step(). A long-context run that passes step 1 comfortably can therefore run out of memory at step 2. The fix is to load offloaded optimizer state only around the optimizer step, and with several updates per step, re-offload after each. The general lesson is to smoke-test with at least two updates at full sequence length.

4 · Judges that fail silently

After a sandbox crash, one judge kept answering, but it graded known-correct solutions as "Wrong Answer": score 0, status done. Retries can't catch that, because nothing looks like an error. I added three fixes:

• A canary: two known solutions with known scores, graded every few minutes. Any deviation pauses training.
• A rule that infrastructure failures are masked, never scored 0.
• A cap on load per judge, since at around 144 concurrent submissions a judge stalled while clients waited up to an hour.

5 · One token over

A trial died three times with "maximum context length is 262144 tokens; you requested … for a total of at least 262145". In every case the request was exactly one token over. The harness estimated prompt size at about four characters per token, but one tool result was a 22K-character output, mostly digits, and this tokenizer emits one token per digit. The 13K-token underestimate exceeded the harness's safety margin. The server then clamped the output budget to the space left but counted one more prompt token in a later check, so the request failed by exactly one token, and failed again the same way on every resume.

6 · Red-team the grader before the policy does

Every grader I built or adopted got an adversarial audit before it was allowed to produce reward. Every audit found something. No policy found these first, but an RL policy optimising against the grader eventually would.

The fixes are boring: score from a channel the submission can't write to; have the privileged grader pick every path itself; time code outside the submission's interpreter; add tamper checks (a reported pass count larger than the number of collected tests zeroes the trial); and grade final numbers by the benchmark's own published rules.

The work itself was done collectively. A coordinator agent and a rotating set of builder, auditor and red-team agents, all running in Claude Code, wrote the harness, ran the experiments and reported back while I made the calls. That's a small preview of the subject, and it taught me a few rules I now apply to every run:

• A validity table on every check-in. KL, entropy, gradient norm, the trainer-vs-inference log-prob gap, truncation rate, the fraction of groups with any learning signal, speculative-decoding acceptance, prefix-cache hit rate, degenerate outputs, and the judge canary. A green dashboard has to be green on all of them.
• Pre-flight from artifacts, not summaries. The checklist is checked against the actual files and logs, not against an agent's description of them. That is how the all-zero 16K run was caught in pre-flight, before it burned real compute.
• Speed first, then go. Push serving and training speed until the next improvement is under about 5%, without changing the recipe, then launch.
• No ablations without a hypothesis. Use one best-known recipe per arm. The first steps of the real run are the pre-flight.
• Red-team every build, then read the diff. Every audit found a real blocker, and the grader exploits in §5 were all caught this way.
• Recount every claim. The coordinator withdrew or corrected more than two dozen of its own claims over four days after recounting against data, including a KL diagnosis, an overstated drop rate, and the expected benefit of a design change. That's the system working, not failing.
• Read trajectories. The 27-token thinking and the self-deleted solutions in §3 came from reading, not from metrics.
• Agents stall quietly. Background agents can sit for hours on a prompt waiting for approval. Give them scripts instead of long inline commands, and watch when their output files last changed.

The question is whether post-training can teach one shared policy to work as a team: when to fork a copy of itself, what to share and with whom, whose result to trust, and when the team is done. Three things set this apart from simply sampling more:

• The harness has no organisation chart. Any agent can fork a child that inherits its full context and an exact copy of its working tree. Agents message each other directly, share an append-only whiteboard, can read each other's recent actions, and any agent can submit. These are the v2 primitives behind the "collective v2" column in Table 1.
• One reward for the whole team: the task's own verifier score. Forking, messaging and agreeing are never rewarded directly, because that would teach ceremonial communication and let plausible errors spread.
• The control decides the claim. The same starting checkpoint is trained two ways, as a solo agent and as a team, with matched training tokens. Both are then evaluated solo, as independent parallel samples with a selector at the end, and as an online team. A team-specific gain only counts if team training improves the online team more than solo training does, beyond what extra parallel samples give.

Credit assignment is the hard part: one team reward says little about which fork or message helped. The tool below exists for that. So far it is a measured tool, not a learning result.

The credit tool: freezing a running team

A team-outcome reward can't say which fork or message helped. The tool I built for that freezes the whole team just before one decision, samples alternatives from the same state, runs each to the end, and credits the decision with the difference (a leave-one-out baseline, as in counterfactual credit-assignment work). With the 9B model on a spec-to-repo task, the same fork decision led to 2, 1 or 151 of 211 passing tests across three branches. The rest of the team was re-sampled in each branch, so the spread includes downstream noise; an earlier run of the same decision gave 48, 24 and 113. Even so, one decision showed a spread large enough to be worth crediting.

Freezing a team means capturing more than text. It includes every agent's context and pending request, each working tree and the container filesystem, the message queues (undelivered messages too), the whiteboard and the timers. The costs that matter:

• Freezing is cheap. The snapshot takes about a second and restoring an agent to its first model call about five. Alternatives start as soon as the snapshot is committed, not when the original trial ends, which in one case was 550 s later.
• Run alternatives in parallel, each on its own server. A decision then costs its slowest branch (374 s), not the sum (1,020 s), and a dedicated server per branch raised throughput 52%, partly because that run's contexts were shorter.
• Route a whole team to one server. Branches share almost their entire prompt, so prefix-cache hits are 97–98% and computed prefill is 2–3% of prompt tokens. Random routing would have cost 57–114% more prefill.
• Pick decisions carefully. Each branch still generated 130–145K tokens, and a FrontierCS branch gave no signal at all because the judge timed out on all three alternatives.

Getting the measurement right took an audit. The first version sampled the original decision from the request built before the freeze and the alternatives from the one rebuilt after restoring, which had one extra resume message. Now every alternative, including the original, restarts from the snapshot and resends a byte-identical request.

1. Agentic solo arm, rebuilt. The minimal loop is red-teamed and its eval is running on the base model and the step-40 checkpoint. RL starts once those are in, with a go/no-go rule set in advance: by step 10, reward on revisited training tasks must be rising.
2. Single-call restart from step 40, with an entropy-preserving change that acts even with one update per step (Clip-Cov is the candidate). A larger problem pool comes later.
3. A short team-RL pilot, roughly 5–10% of the full budget, with paired solo-vs-team evals at every checkpoint and freeze-and-branch credit on fork, message and submit decisions.
4. Several attempts per task on every headline comparison. Table 1 shows why one isn't enough.

The next post will say whether a team learns any of this.