Skip to content

About this recipe

Status
tried
Stage
Implement with agents
Output check
Hard to verify
Autonomy
Implements a written specification in small steps; a second person reviews the evaluation
Last reviewed
2026-10-30

Recipe 8

Pin down the method before the agent starts, and watch for drift

Try this first. Write the specification, including the evaluation, before any code, and have a second person read the evaluation code.

Summary

The hardest errors to catch are small changes to a method that still look correct. An agent asked to implement a plan can deliver something close to, but not exactly, what was planned. Such deviations are easy to miss in evaluation code, and they can change results considerably if they accumulate. Familiar engineering habits, such as test-driven development, help when they are applied more strictly.

Steps

  1. Work out a written specification with the agent before any code exists. Include the evaluation protocol: data splits, metrics and seeds.
  2. Discuss the specification with colleagues.
  3. Write the tests before the code.
  4. Let the agent implement in steps small enough to read. When an agent changes too much at once, nobody can follow what happened, which creates technical and cognitive debt.
  5. Ask someone other than the person who prompted the agent to review the evaluation code.

How to check the output

A second person compares the evaluation code with the written specification. Tests written before the code must pass. Each change is small enough to read in full.

Examples

  • An agent asked to implement an analysis for a paper changed how the cross-validation was done. The code ran, the paper's result shifted slightly, and an attentive co-author noticed only much later. It was one of several cases where an agent delivered something close to, but not exactly, what was planned.
  • Groups that build software with agents now write the specification with the agent first, and include the evaluation protocol in it.

Known problems

  • Large pull requests contain changes that no reviewer reads in full.
  • The person who prompted the agent tends to read the code as what they asked for, not as what it does.
  • In the centre's survey, 31% of group leaders had seen an agent error reach a shared result.

Skills

Each skill page has a brief. Give it to your agent to build a version for your own institute.

  • public-data-qc: downloads, deduplicates and checks public datasets, with a report for each step.

Best practices on aiforscience.eu that this recipe follows:

Change history

  • 2026-10-30: Renumbered to follow point 8 of the submitted Comment (formerly "Review methods, limit file access"). The sandbox and folder-permission steps moved to recipe 2.
  • 2026-10-28: Updated to match the presubmission draft of the Comment.
  • 2026-09-25: First draft, from the first round-table.