Recipe 7

Turn open questions into checkable problems¶

Try this first. Before any automated search, fix a metric, a baseline and at least three datasets.

Summary¶

Write the research question as a metric, several datasets and a baseline. Then let an agent try methods: parameter sweeps, rewritten methods, or a search over code variants. The benchmark scores each attempt, so the agent does not need to be told which method to use. Specifying the problem is the hard part: the quality of the problem definition limits the results more than the agent does. Well-posed quantitative problems, such as benchmarks and tasks for virtual-cell models, are infrastructure that institutes should build.

Steps¶

Pick a problem with data you already have.
Write the metric and the evaluation script before the agent starts.
Use at least three datasets, so that a result that only works on one of them is visible.
Run a baseline method, so each result can be compared with it.
Do not give the agent write access to the evaluation code.
Move to harder problems only when results on the first one are stable.

How to check the output¶

The benchmark scores each result. A person checks the benchmark itself: are the splits, metrics and seeds the ones written down in step 2? Is the baseline result correct? A result that improves only one dataset does not count.

Examples¶

On a public single-cell benchmark, agents help to find which parameters of a method matter most.
A published code-search system produced dozens of methods that beat every published entry on the same benchmark's leaderboard.
The centre is inviting its group leaders to add their own methods to a shared environment where agents optimize them against agreed metrics on central computing.
Benchmark communities such as Open Problems in single-cell analysis and CASP for protein structure prediction already define problems this way.

Known problems¶

With one dataset, an agent improved a metric by changing random seeds and small parameters. See What went wrong.
An agent that can edit the evaluation code can change what is being measured.

Skills¶

No skill yet. Contribute one.

Best practices on aiforscience.eu that this recipe follows:

Evaluate tools before trust: build the benchmark before the agent uses it.
Match the method to the task: an agent search only makes sense if there is a metric.

Change history¶

2026-10-30: Renumbered to follow point 7 of the submitted Comment (formerly "Let agents search for methods").
2026-10-28: Updated to match the presubmission draft of the Comment.
2026-09-25: Updated to match the first full draft of the Comment.
2026-09-25: First draft, from the first round-table.