Refinement, RL, and the agent harness: Hermes and Prime Agent
An agent can revise its instructions without retraining its model. Hermes and Prime Agent show how the software around a model can retain useful lessons, organize work, and make improvement measurable.
Start by asking what changes
A coding agent fails a task, reads the error, and succeeds on another attempt. Has it learned? To answer, inspect what will be different the next time it works: the current answer, a saved procedure, or the model’s learned parameters.
The harness is the software around the model. It supplies tools, assembles context, runs actions, records results, and manages continuing work. Changing those parts can change an agent’s behavior while keeping the same underlying model. The Continual Harness research treats this external state as something an agent can revise from experience.
This article distinguishes three improvement mechanisms and one execution pattern. The descriptions of Hermes Agent and Prime Agent reflect their documentation checked on 15 September 2026. The examples are illustrative; this is not a hands-on comparison of the two tools.
| Mechanism | What changes? |
|---|---|
| Answer refinement | The current draft or attempt changes after feedback. |
| Harness refinement | Saved instructions, skills, memories, or other operating state change. |
| Model-training RL | Training updates the model’s parameters, or trainable adapters, using reward signals. |
| RLM execution | The agent organizes context and model calls programmatically. This does not itself require a training update. |
Sources: Continual Harness (2026)
Refinement: turn feedback into a better next attempt
Refinement starts with an attempt and feedback about it. The agent uses that feedback to propose a revision, then tries again. The feedback might come from a compiler, a test, a person, or another model call.
The 2023 Self-Refine paper studied a particular version: one language model generated an answer, critiqued it, and revised it without additional training. That is answer refinement. Saving a reusable lesson adds another step: the improvement can then influence later work, if the harness loads it when relevant.
For a useful saved lesson, prefer an observed cause and a check over a vague instruction to be more careful. A test failure can identify a missing case; a fluent self-critique may only supply a plausible explanation. Treat a proposed lesson as a hypothesis until another attempt supports it.
A practical harness-refinement loop
- ObserveInspect the attemptKeep the task, actions, errors, and checks that explain what happened.
- ReviseSave a narrow lessonChange a relevant instruction or skill. Record its scope and supporting evidence.
- CheckTry a fresh caseLoad the revision and measure the next result. Keep, adjust, or roll it back.
Feed the next result back into observation. The model’s trained parameters can remain unchanged throughout.
Sources: Self-Refine: Iterative Refinement with Self-Feedback (2023)
Reinforcement learning: train from the consequences of actions
In model-training reinforcement learning, or RL, a policy is the model’s rule for choosing outputs and actions. A rollout is a recorded attempt: the observations, choices, tool interactions, and resulting outcome. A reward assigns a score to some part of that behavior or to its final result.
A training algorithm uses those scored attempts to update trainable parameters. In policy-gradient methods, the optimizer estimates which choices should become more or less likely to improve expected reward. PPO is one example of this family; it alternates collecting interactions with optimization steps. Different training systems use different objectives and ways of assigning credit.
The reward may arrive only after many actions. A high score does not identify every earlier choice as correct. That credit-assignment problem is one reason RL involves a training procedure, not simply telling an agent that it did a good job.
The model-training loop
- CollectRun the policyThe current model attempts tasks in an environment. Record its actions and outcomes.
- ScoreCompute rewardsA grader evaluates the behavior against the task’s success criteria.
- OptimizeUpdate parametersThe training algorithm adjusts weights or adapters, then collects more attempts.
Evaluate the trained version on separate tasks. An increase in training reward alone does not establish broader improvement.
Sources: Schulman et al., Proximal Policy Optimization Algorithms (2017); Prime Intellect’s prime-rl training framework
Follow one mistake through both loops
Imagine an agent writing a CSV importer. It splits each row on commas. A test with a quoted address containing a comma fails because the importer creates an extra field. This is a hypothetical example.
Answer refinement fixes this attempt: use a CSV parser and rerun the test. Harness refinement retains a scoped procedure: when implementing CSV ingestion, use a parser that handles quoted fields and test embedded delimiters. In a later task, the agent can read that procedure without any model retraining.
An RL experiment would be different. A trainable model would attempt a collection of parsing tasks, receive scores from protected correctness checks, and undergo parameter updates. To find out whether it improved, evaluate on reserved cases it did not train on. Keep those cases out of both the reward-driven updates and the saved lesson’s development.
The distinction tells you where to inspect a failure. Did the lesson never get saved, fail to load, give bad advice, or reach a model that ignored it? Those require different fixes from a training run that rewards the wrong behavior.
Hermes: give experience a reusable home
Hermes Agent, from Nous Research, separates concise persistent memory from reusable skills. Its memory documentation describes notes and user information loaded as a snapshot at session start. Writes made during the session persist to disk; the injected snapshot refreshes at the next session boundary. Tool responses can expose the current state sooner.
Its skills system loads procedure documents on demand. The agent can create and update skills, and supporting references can be opened when needed. That gives a procedure a discoverable home without putting every instruction into every model call.
These are useful design choices for recurring work. A preference belongs in a short memory; a multi-step procedure needs instructions and verification. My assessment is that this separation makes Hermes worth studying as a harness: it provides concrete places to inspect what was retained and how it becomes available again. Its usefulness still depends on the quality of the saved material and whether the agent selects it.
Hermes Self-Evolution is a separate optimization project
Nous Research also maintains Hermes Agent Self-Evolution. It uses DSPy and GEPA to propose skill revisions, evaluate candidates, and select improvements through reflective search. This works on the agent’s external material through model API calls; it does not require training the model’s weights.
The repository’s status table matters more than its broad ambition. At the time checked, skill-file optimization is marked implemented. Tool-description, system-prompt, and implementation-code optimization, plus a continuous improvement pipeline, are marked planned. The repository also describes test gates and human review before changes are merged.
This is an additional project with its own setup and evaluation process. Ordinary Hermes use should not be described as automatically running that whole pipeline, or as continuously applying RL to the model behind the agent.
Sources: Nous Research: Hermes Agent Self-Evolution, workflow and implementation status
Prime Agent: make context and delegation programmable
Prime Agent’s RLM means recursive language model, not reinforcement learning. Its programming model gives the agent a persistent Python environment. The agent can manipulate data, inspect files, run commands, and delegate focused work to child agents through code. Working variables can survive across tool calls and context compaction.
For example, an agent investigating a large log could keep the parsed records in a variable, filter them programmatically, and ask a child agent to examine one suspicious sequence. The parent can inspect the relevant result without loading the entire log into every model call. This is an illustrative use of the documented interface.
That design is attractive for work involving large intermediate results or several related investigations. It gives the model a way to organize computation outside its immediate context. It also creates work to manage: the parent must ask useful questions, inspect returned evidence, and account for the cost of its children. More delegation is not itself a measure of success.
Sources: Prime Agent: RLM Programming Model
What Prime Agent’s /refine retains
Prime Agent documents /refine as a review of the current trajectory: the recorded sequence of work. It can save focused changes to supplemental prompts, memories, skill descriptions, and reusable subagent specifications. The immutable base system prompt remains unchanged, and recorded snapshots support rollback.
The documented default is session-local refinement. Do not assume that a lesson saved in one session becomes a universal instruction for every future task. Scope and later loading are part of the mechanism, not administrative details.
The useful combination is a programmable working environment plus revisable operating knowledge. One helps the agent perform an investigation; the other can preserve a procedure discovered during it. Neither operation, by itself, retrains the language model.
Sources: Prime Agent: Continual Harness and /refine behavior
How harness refinement and RL can work together
The loops can be combined when a research or training system is built to do so. The Continual Harness paper describes both external harness adaptation and an experiment that uses scored agent trajectories to update an open model. That is evidence for a specific combined research setup, not a default behavior of every application using the idea.
Prime Intellect’s wider tooling also makes the connection explicit. Environments package tasks, agent interaction, and scoring for evaluation or RL. Its separate prime-rl framework supports model training with verifiers environments. Prime Agent, an interactive harness, and prime-rl, a training framework, have different jobs.
The engineering opportunity is to reuse trustworthy task feedback. First make it possible to observe whether work succeeded. Then compare an instruction change, a model update, or both. If the harness and the model change together, record both versions so you can later investigate which change helped.
Sources: Continual Harness: combined harness and model adaptation; Prime Intellect: Environments Hub; prime-rl: training and evaluation
A strong harness still needs evidence of improvement
The Prime Agent technical report includes long-running experiments and comparisons under stated task budgets. Its Factorio discussion is especially instructive: the authors report sustained factory progress, but also describe a different trace that saved a resource-spawning exploit as a reusable skill. Persistence retained an unwanted shortcut.
That example shows why the feedback mechanism matters. A refinement process can preserve a bad lesson; a reward-driven trainer can optimize a flawed score. The report is evidence about its authors’ experiments. It does not establish that either harness will improve every reader’s workflow, and we have not independently reproduced those results.
A practical evaluation can start with the CSV example above. Use this checklist as an experimental design, not as a completed benchmark:
- Hold the model, tool access, task inputs, and budget fixed when testing a harness change. Record the exact versions and saved state.
- Compare the original harness with the refined version on fresh cases. Reset unrelated session history so it cannot supply the answer accidentally.
- For an RL comparison, evaluate the original and trained model under the same harness. Keep final evaluation cases outside the training process.
- Record task correctness, tool failures, tokens, elapsed time, and human interventions. Report repeated attempts, including failures.
- Protect the grader and inspect the resulting artifact. A high score obtained by weakening a check is not the intended improvement.
- Try removing the saved lesson or reverting the model update. This helps test whether the claimed change explains the observed difference.
Hermes and Prime Agent are worth studying because their documented designs make experience available for later decisions and give agents useful ways to act on it. The strongest case for a harness is a traceable change that improves the next task under a fair comparison. Start with the feedback, identify what gets updated, and check what the next attempt actually does.
Sources: Prime Agent technical report, evaluation and Factorio traces (§3)