Recipe 6

Let the cost of checking decide how much freedom an agent gets¶

Try this first. Start with tasks a machine can check, such as tests that must pass or numbers that must match.

Summary¶

The freedom an agent gets should depend less on how hard a task is and more on how cheaply its output can be checked. The clearest successes came where checking was nearly free. Analysis, experimental design and writing are much harder to check, so agents get less freedom there: they draft, and people check almost everything.

Give agents more freedom in three steps:

Tasks that a script can check.
Running experiments on tested code that the agent cannot change.
Writing new code, with a person reviewing every change.

Move to the next step only when the current one works.

Steps¶

Tag a release of tested code and give the agent read-only access to it.
Let the agent start GPU experiments, collect the results and keep a log of each run.
Let an agent watch cluster jobs, read the logs of runs that crashed, fix trivial errors, resubmit, and log each change. Set a limit on the number of resubmissions.
For code that is rewritten to run faster, such as GPU kernels for large training runs, let the agent test and refine it in a loop. Each version must return the same numbers as the reference code.

How to check the output¶

In the first two steps a script does the check: the code is unchanged, the rewritten code returns the same numbers, and the run log lists each resubmission and its cause. In the third step a person reviews each change before it is merged.

Examples¶

One group lets agents run thousands of experiments on fixed code. Within those limits, the agent found better model settings than the group had found by hand.
With an agent monitoring its cluster jobs, one group went from tens of runs to hundreds.
In protein language model training, an agent implements about ten published methods a day. A benchmark scores each one, and a person decides which to keep.

Known problems¶

An agent that resubmits jobs can use up a shared quota. Set a limit on the number of runs.
When agents move from running code to writing it, they can change a method without anyone noticing. See What went wrong and recipe 8.

Skills¶

Each skill page has a brief. Give it to your agent to build a version for your own institute.

job-babysitter: watches cluster jobs, resubmits failed runs, and logs each change.

Best practices on aiforscience.eu that this recipe follows:

Govern agent autonomy and accountability: read-only code limits what the agent can do.
State human-in-the-loop requirements: each of the three levels below has a different human check.

Change history¶

2026-10-30: Renumbered to follow point 6 of the submitted Comment (formerly "Run experiments on fixed code").
2026-10-28: Updated to match the presubmission draft of the Comment.
2026-09-25: Updated to match the first full draft of the Comment.
2026-09-25: First draft, from the first round-table.