Single-Rollout Asynchronous Optimization for Coding Agents: Practice in Lego-RL

TL;DR

Coding-agent RL often samples multiple rollouts for the same task, each involving a long sequence of searches, edits, and tests. Can one rollout provide enough signal to learn?

We study Single-Rollout Asynchronous Optimization (SAO) in Lego-RL, using a learned critic to train from one rollout per task. We compare it with GSPO on OpenSWE and test multilingual bugfix tasks and mixed-harness training.

SAO makes single-rollout training practical. It improves on the base model with 73.7% fewer trajectories to its own best checkpoint than GSPO, though GSPO scores higher on Verified. Critic warmup and advantage whitening are central to the recipe.

Why train with one rollout per task?

Group-based optimization makes expensive coding rollouts depend on one another. Each rollout can involve dozens of searches, edits, and tests, and GRPO needs several rollouts for the same task to compare their rewards. Our GSPO baseline uses these group-relative advantages with sequence-level policy ratios. Asynchronous generation keeps other tasks moving, but completed rollouts still wait for their group and become older relative to the policy.

SAO removes this group dependency by learning a critic. Its values support token-level generalized advantage estimation (GAE) from one rollout per task, while direct double-sided importance sampling masks tokens whose current-to-rollout probability ratios fall outside a prescribed interval. The tradeoff is a new learning problem: the critic must provide a useful baseline without same-task alternatives.

Sample counts refer to these experiments. Both methods can train on many tasks in each batch.

Our training recipe freezes the critic’s attention parameters and adds two critic epochs per actor update. GAE covers model-action tokens; tool observations remain in context.

How does SAO compare with GSPO?

We start from Qwen3.5-35B-A3B and train on 2,699 OpenSWE tasks through OpenHands SDK. Both methods use fully asynchronous training on the same machines. GSPO samples eight rollouts per task; SAO samples one.

SAO improves on the base model across all three benchmarks, with a different balance of gains from GSPO. GSPO leads on Verified and narrowly on Pro; SAO scores higher on Multilingual. These results establish that one rollout per task can support learning, while leaving a gap on Verified.

Table 1. Resolved tasks (%) through OpenHands SDK. Parentheses show gains over the base model in percentage points (pp).

SAO’s clearest saving is in sampled trajectories: it uses 12.8k to reach its own best checkpoint versus 48.6k for GSPO, a 73.7% reduction. The corresponding validation scores are 68.6% and 70.4%, so this is not a comparison at matched quality. SAO’s step time is 5.4× shorter: reducing rollouts per task from eight to one cuts each batch from 512 to 64 trajectories, while critic value inference and two critic-training epochs per actor update add overhead that partly offsets the sampling savings.

Figure 1. Sampling cost and training time, relative to GSPO = 100. Best-checkpoint panels use each method’s own best score.

Reward rises for both methods, but GSPO’s trajectories grow longer and include more turns, whereas SAO’s response length grows modestly while its turn count falls. One plausible contributor is length normalization: Dr. GRPO shows that it weakens per-token penalties on long incorrect responses, favoring their persistence. GSPO’s gradient also combines a shared sequence advantage with an inverse-length factor, so a similar mechanism may contribute here.

Figure 2. Training progress by Step or Hour. Training curves default to trailing ten-point means; validation is unsmoothed.

What makes SAO work?

Critic preparation is central to making single-rollout training work. We keep the base actor, warm-start the critic from a trained checkpoint, and extend critic-only warmup from 10 to 20 steps so its values can adapt to the actor’s rollouts before policy updates begin. Under the stabilized recipe, mean critic explained variance rises from 0.0 to 0.3 over the common steps 40–100. Warm start, warmup, and whitening change together, so this comparison evaluates their combined effect.

Advantage whitening controls the actor’s training signal while the critic is still imperfect. We enable GAE_WHITEN_ADVANTAGES=True to center and scale raw advantages before policy updates:

Here, μ and σ² are computed over all valid model-action tokens across the training batch, excluding tool observations. Every trajectory shares these statistics; longer trajectories contribute more tokens. Whitening changes only the actor’s advantages, preserving the critic return targets computed beforehand.

Figure 3. Initial and stabilized recipes, with Training and Critic & actor views. Curves default to ten-point means; validation and advantage means are unsmoothed. Explained variance uses a symmetric-log scale; critic gradient norm uses a log scale.

The initial recipe suffered policy collapse: repetitive think tool calls crowded out editing and execution, reaching 73.0% of tool calls around step 130. With a weak critic and whitening disabled, three consecutive all-negative advantage batches preceded this surge. The stabilized recipe avoids this pattern, keeping think calls near 6.0–8.0%. Best validation rises from 66.4% to 68.6% over a longer run, while mean turns fall from 109.5 to 49.1 at common steps 120–139.

Does SAO work beyond OpenSWE?

SAO’s gains extend to multilingual bugfix tasks and mixed-harness training. Both settings start from Qwen3.5-35B-A3B, use a trained critic and advantage whitening, and evaluate through OpenHands SDK on 300 SWE-bench Multilingual tasks. Keeping the evaluation harness fixed lets us assess both training settings through the same interface.

Table 2. Additional SAO settings. Data, batch size, critic initialization, and training budget also differ, so this comparison does not isolate the effect of mixing harnesses.

Figure 4. Multilingual training curves with ten-point means. Validation is unsmoothed; stars mark best checkpoints.

Both settings improve from 51.7% at step 0. OpenHands SDK training peaks at 57.0% and ends at 54.3%, with reward, response length, and turns all increasing. Mixed-harness training samples OpenHands SDK, Claude Code, and OpenCode at equal configured weights; it peaks at 61.7% and ends at 57.0% (57.6% across the last five evaluations). The data and training settings also differ, so the higher mixed-harness score cannot be attributed to harness mixing alone.

What does a useful critic see along a trajectory?

A useful critic should track how the prospect of success changes within a trajectory. We inspect 40 pre-update OpenHands SDK traces—one sampled success and one failure at each of twenty training steps—covering 959,418 model-action tokens. Tool observations remain in context but are omitted from the token axis, so the plots show values at the agent’s own actions.

The two examples illustrate the desired progression: values stay near one and rise later in the successful trajectory, while falling toward zero in the failed trajectory. The success example moves from 0.9 to 1.0 with a higher second-half mean; the failure moves from 0.2 to 0.0 with its second-half mean near zero. We summarize this change using ΔV = V[last token] − V[first token], alongside first- and second-half means.

Figure 5. Token examples and summaries of all 40 traces. Faint curves show token values; thick segments show turn means. Dotted lines mark turn boundaries; dashed lines show return targets. The examples use different vertical scales. Switch views for mean values and last-minus-first differences; zoom to inspect tokens.

Values can fluctuate or exceed the target range because critic outputs are unconstrained; monotonic trajectories are not required. Success traces have higher mean values than paired failures in 19 of 20 cases. Selection by outcome means this is not held-out classification accuracy.

Large value jumps cluster at turn boundaries, where fresh tool feedback can change the critic’s estimate. Across 3,450 verified message-boundary transitions and 955,928 within-turn transitions, changes with |ΔV| > 0.1 occur in 1.9% versus 0.2%, about 9.2× as often. The highlighted examples jump by +0.3 and −0.1 at their respective boundaries. This pattern is consistent with the critic responding to new context, although explaining any individual jump requires the interaction log.

What should we take away?

SAO makes one rollout per task a practical way to train coding agents. In Lego-RL, it improves all three benchmarks over the base model and reaches its own best checkpoint with 73.7% fewer trajectories than GSPO. The multilingual and mixed-harness results extend this recipe to different tasks and training interfaces.

Making that recipe work starts with the critic: warm-start it, give it time to adapt before updating the actor, and whiten advantages across the training batch. Then follow validation, trajectory length, and token-level values together to understand both what the agent learns and how its behavior changes.

Citation

If you use Lego-RL, please cite the Lego-RL paper.

@article{du2026legorlharnessnativereinforcementlearning,
  title = {LEGO-RL: Harness-Native Reinforcement Learning for Coding Agents},
  author = {Yiming Du and Yuxin Jiang and Tao Yuan and Jianbo Dai
            and Shaowei Wang and Jierun Chen and Chaofan Tao
            and Xianzhi Yu and Lifeng Shang and Kam-Fai Wong
            and Xiaohui Li and Haoli Bai},
  year = {2026},
  journal = {arXiv preprint arXiv:2608.17393},
  url = {https://arxiv.org/abs/2608.17393}
}