This document describes the algorithm implemented in src/simple_scibuddy/. It explains how scientific interactions guide harness changes, how the selected harness supplies fresh experience for model training, and how the updated model returns to the next harness stage.

Overview · Inner recursion · Outer recursion · Execution contract · Evaluation · Handoff and continuation · Code map

For runnable configuration and commands, see the experiment guide. For the researcher-facing product and demonstration, see the ScienceBuddy overview.

The learned system has two components:

• A task model $M_k$ generates the agent's responses and tool-use decisions.
• A harness $H_{k,j}$ is a Python program that organizes prompts, model calls, tool execution, context and answer submission.

Here $k$ indexes model-learning cycles and $j$ indexes harness updates within one cycle. An independent auxiliary model proposes harness changes. Its parameters remain fixed while the task model and harness evolve.

The inner recursion fixes $M_k$ and searches for a better harness. The outer recursion fixes the selected harness and updates $M_k$ through reinforcement learning. The selected harness is inherited by the next cycle:

$$ H_{k,j+1} = \mathrm{Select}(H_{k,j},\ \mathcal{C}_{k,j};\ M_k,D_{\mathrm{val}}), $$

$$ M_{k+1} = \mathrm{GRPO}(M_k;\ H_{k,J},D_{\mathrm{train}}), \qquad H_{k+1,0}=H_{k,J}. $$

$\mathcal{C}_{k,j}$ denotes the candidates proposed from the current interaction evidence. $J$ is the number of harness steps per phase. A change in the harness changes how the model collects experience; a model update changes which harness procedures are effective in the next phase.

Original figure from the ScienceBuddy paper. Panel B depicts the coupled learning processes. The figure also covers the broader product's task construction, adaptive environments and online deployment. The implemented experiment uses a frozen task release and sequential harness/RL stages.

The default schedule has three complete cycles:

(M0, H0)
   → h0001: harness learning → r0001: model learning → M1
   → h0002: harness learning → r0002: model learning → M2
   → h0003: harness learning → r0003: model learning → M3

It ends after r0003; it does not add a fourth harness stage. Each harness phase has three steps and each RL phase has 30 updates in the maintained recipe.

At phase start, the inherited harness passes a training-task execution preflight and receives a baseline Test evaluation. Neither Test outputs nor Test labels become proposal evidence.

For each harness step, the current program runs on the next 16 Train tasks. The Train ordering is determined from the configured seed and task IDs. A cursor continues across phases; it wraps only if the training index is exhausted.

An interaction can contain several model calls and Python tool executions. It can also contain up to one simulated researcher reply in the current recipe. The episode's reported reward remains the verifier score of its first submitted answer, even if a follow-up leads to a corrected answer.

The experimental feedback protocol is verifier-assisted:

1. The host grades the submitted answer using the private reference.
2. The host constructs an allowed set of researcher replies. The alternatives distinguish acceptance, invalid answer format, a recent tool error and a generic request to reconsider.
3. The auxiliary model selects a reply from that allowed set. Invalid output falls back to an allowed template and is recorded as such.
4. A PRM-style interpretation request relates the reply to the public execution record: model outputs, tool observations, the submitted answer and the reply.

The PRM labels the feedback and identifies relevant fields. Host validation checks its structure, field addresses and feedback classification. Quote text is not required to match verbatim. Interpretations remain hypotheses: a generic correction does not establish a particular scientific cause, and acceptance does not validate every intermediate step.

The proposal evidence contains public questions, actual model inputs and outputs, tool calls, observations, valid feedback annotations and execution diagnostics. Private reference answers and numerical grading outcomes are excluded. Invalid PRM annotations do not remove the underlying public execution record.

The current step's evidence is combined with a bounded selection of earlier public failures/corrections and successful controls from the same phase. When a candidate is adopted, pending evidence is cleared so the next search reflects the new harness. Aggregate proposal history persists across model updates.

Each step requests three independent candidate hypotheses from the same parent. All proposals are generated before the candidate Val results are revealed. Each candidate can replace the complete run(task, api) program, including prompts, parsing, control flow, memory and tool orchestration.

The improver receives the parent program, public resource inventory, execution patterns, feedback interpretations, modification history and the task model's budgets. A proposal must cite evidence from at least two distinct Train tasks.

The structured command interface supports:

The current protocol permits at most three reads, two repair attempts and six auxiliary calls per candidate. Reads expose the selected public field from the requested offset; preview truncation is not a substitute for reading the field. A repair to an already parsed candidate does not reopen evidence reads.

The host parses candidate Python without executing it. Structural validation requires a synchronous run(task, api) interface and rejects known ineffective instruction patterns. Execution occurs in an isolated controller container.

A structurally valid candidate must first execute on the designated Train preflight task. Preflight checks execution and answer-format validity; it does not require a correct scientific answer. Byte-identical programs are excluded.

The parent and each remaining candidate are then evaluated on all 90 Val tasks:

• One first-answer attempt per task, without researcher feedback.
• Temperature 0, matched task seeds and the same host budgets.
• Complete task-ID coverage for every program being compared.
• One shared concurrency limit across candidate evaluation jobs.

For a fixed model $M$ and harness $H$, define:

$$ S(H;M,D_{\mathrm{val}})=\frac{1}{|D_{\mathrm{val}}|} \sum_{x\in D_{\mathrm{val}}} \mathbf{1}!\left[\mathrm{Verify}_x(a^{(1)}(M,H,x))=1\right]. $$

$a^{(1)}$ is the first submitted answer. A candidate is eligible to replace the parent only if its complete Val evaluation has no candidate runtime errors and its score is strictly higher than the parent's score. Here errors refer to recorded candidate-program failures (episode.error); an ordinary tool error or exhausted episode budget is a separate recorded outcome, not automatically a controller failure.

Selection retains the parent on a tie. If several candidates tie above the parent, the earlier candidate wins. The parent is evaluated again at every step rather than assigned a cached score from a previous comparison.

The selected score, parent score, candidate validity and decision are recorded. Only aggregate Val comparisons enter subsequent proposal history; Val traces do not become improver evidence. At the final harness step, the selected program is evaluated on Test and handed to the outer stage.

The selected harness is frozen for the entire RL phase. SkyRL samples eight task instances per batch and eight attempts per instance, giving 64 episodes in the default recipe. N_SAMPLES_PER_PROMPT=16 changes the attempt count independently of rollout concurrency.

These are fresh on-policy attempts generated by the current task model. The harness-stage conversations supply procedural evidence, not replayed policy completions for RL. Val tasks are excluded from the RL training index.

Training uses temperature 1 and top-p 1 with thinking disabled. The first answer submission ends the episode; no simulated researcher reply is provided during RL.

The reward is the host verifier's binary outcome for the submitted answer:

$$ R_i=\mathrm{Verify}_{x_i}(a_i^{(1)})\in{0,1}. $$

The verifier parses the required answer format and checks the private reference. An attempt that never submits an answer has no successful outcome. PRM feedback labels are not substituted for this reward.

Each model call becomes a training row containing its actual prompt token IDs, completion token IDs and rollout log probabilities. Calls in one episode share a trajectory identity. The episode reward is placed once, on the final completion; earlier calls receive no separate terminal reward.

Only model-generated completion tokens are training targets. Prompt tokens, tool outputs, simulated researcher text and generated harness Python are not optimized as model outputs. An episode with no model calls contributes a recorded failure but cannot supply a policy-gradient row.

For attempts $i=1,\ldots,G$ on the same task instance, GRPO compares their terminal rewards with the group's mean and standard deviation:

$$ \mu_x=\frac{1}{G}\sum_{i=1}^{G}R_i, \qquad A_i=\frac{R_i-\mu_x}{\mathrm{std}(R_1,\ldots,R_G)+\delta}. $$

The pinned SkyRL implementation provides the numerical stabilizer $\delta$ and uses standard-deviation normalization by default. A group with multiple valid attempts and identical rewards has zero relative advantage. A group containing both outcomes provides positive and negative relative advantages. The backend has a separate fallback for a single surviving trainable trajectory; zero-call failures supply no training row, and an entirely empty batch is rejected.

Step-wise trajectory handling computes the scalar advantage from each episode's last model call, then broadcasts it to that episode's model-call rows with their response masks. An episode with several tool-use steps therefore remains one rewarded attempt rather than several independently graded tasks.

The regular clipped policy objective compares the updated policy with the recorded rollout policy at generated tokens. Writing this token ratio as $\rho_{i,t}(\theta)$, the surrogate has the form:

$$ \min\left(\rho_{i,t}(\theta)A_i, \mathrm{clip}(\rho_{i,t}(\theta),1-\epsilon_{\mathrm{low}},1+\epsilon_{\mathrm{high}})A_i\right). $$

The pinned backend controls clipping and loss reduction; its default reduction is a mean over valid tokens. The maintained recipe uses a learning rate of 1e-6, one update epoch per batch, and disables both KL loss and KL reward penalties. The default stage has 30 optimizer updates.

After the final update, the worker exports model weights, configuration and tokenizer files. It loads that export in a separate evaluation process and checks complete Test coverage and nonempty model generation before accepting the stage. A checkpoint directory or a successful training-process exit is insufficient on its own.

The verified export becomes $M_{k+1}$, and the selected Python program becomes $H_{k+1,0}$. Optimizer/trainer state is not retained in these exports. The next stage starts from the exported model weights.

The generated harness can organize the agent's behavior, but the host retains control of the scientific environment and training contract:

The controller and execution containers have no network access. Only the scientific execution container receives the public task assets and frozen data lake. Private references and service credentials stay on the host.

Messages identify their source as task, model, tool, feedback or harness. The host validates this metadata, applies tool-history budgets and removes the metadata before tokenization. Task questions and feedback must match their actual source; user-role harness instructions and budget notices must be explicitly identified.

The host clips tool responses and older tool history within the configured token budgets. Clipping preserves recorded provenance and the actual shortened model input. A new harness cannot escape these budgets by renaming a tool observation.

Candidate controller errors make a program ineligible. A confirmed memory failure in the scientific execution container stops the affected episode with zero reward. Unrecognized solver/runtime infrastructure failures remain explicit errors rather than scientific evidence for a new procedure. Auxiliary proposal failures are recorded as failed or skipped proposal attempts.

The three splits serve different purposes:

The same Test assignment is used in both learning phases. Task IDs are disjoint, but material groups overlap across Train–Val and Train–Test as declared in the frozen release. This is not a claim that the Test material is independent or previously unseen.

The maintained always_debug trigger runs the fixed stage schedule whether or not the harness improved its Test score. The optional improvement trigger uses the stage's changed harness and Test gain to control handoff; reporting an unbiased final assessment under that alternative requires a separate final holdout.

The default measurement schedule is:

1. Harness stage baseline Test evaluation.
2. Full Val comparison at every harness step.
3. Harness stage-end Test evaluation of the selected program.
4. RL baseline Test evaluation under the inherited harness.
5. Test evaluation from the actual stage-end model export.

During co-evolution, the trainer suppresses post-update in-trainer evaluation so the worker's export-load evaluation supplies the stage-end measurement. The standalone model mode can use its configured periodic evaluation schedule.

Training accuracy measures changing sampled batches. Val scores are used for selection. Neither should be presented as an independent Test curve. Report first-answer accuracy, attempt budgets and task denominators together; pass@k coverage is a different metric and requires the corresponding repeated attempts. Use only recorded measurements for results, and retain the original run provenance when a stage was inherited through continuation.

The experiment pins the identities of the dataset, verifier, runtime, model and harness. A harness stage publishes an immutable request containing the selected program and RL execution contract. A worker claims the request, trains the model, validates its export and publishes a matching result. The next stage consumes only a validated result.

Continuation always starts a new run directory. It can resume from a completed RL stage using its verified model export. If RL failed after a fully completed harness phase, it can reuse that harness and restart the RL stage from the pinned base model. Unsaved optimizer updates are not restored or counted as part of the new model's learning history.

The continuation validator checks experiment settings and dataset/runtime/verifier identities, the selected harness hash, the published handoff, and the appropriate base checkpoint. A still-running source experiment cannot be restarted through the failed-stage recovery path.

model, harness = M0, H0
history, train_cursor = empty, 0

for cycle in 1..3:
    keep model fixed
    preflight inherited harness; measure baseline Test
    pending_evidence = empty

    for step in 1..3:
        collect 16 Train interactions with current harness
        advance train_cursor; accumulate public evidence
        propose 3 candidates from the same parent and evidence
        preflight valid, distinct candidate programs
        evaluate parent and valid candidates on all 90 Val tasks
        adopt the earliest strictly better error-free candidate, or retain parent
        record the selection and aggregate Val history
        clear pending_evidence after adoption

    measure selected harness on Test
    publish immutable model/harness/runtime handoff
    release harness-stage inference services

    keep selected harness fixed
    measure RL baseline Test
    for update in 1..30:
        collect fresh grouped Train attempts
        compute first-answer verifier rewards and GRPO advantages
        update task-model parameters from actual generated tokens

    export model; load and evaluate export on Test
    publish and consume validated stage result
    inherit selected harness and exported model for the next cycle

This pseudocode describes a fresh run with the fixed always_debug schedule. Continuation and the optional improvement-triggered stop follow the rules above.

GRPO advantage computation and clipped policy optimization are delegated to the pinned upstream code in skyrl/, including skyrl/train/trainer.py and skyrl/backends/skyrl_train/utils/ppo_utils.py within that submodule. The owned adapter preserves their step-wise trajectory contract.