`job-babysitter`¶

Watches cluster jobs, resubmits failed runs up to a set limit, and logs each change.

Recipe: Let checking cost set freedom
Output check: Machine-checkable The log lists each resubmission and its cause. The code and parameters of each run are unchanged, which a diff confirms.
Status: brief

Make your own version¶

This skill depends on local details, so the site publishes a brief instead of one institute's skill. Copy the brief and give it to your agent. If your agent has a skill-creation skill, such as Anthropic's skill-creator, it will use it. The agent asks you the questions under "Local details", builds the skill, and runs it once on the test case.

An agent that can read web pages can also fetch the brief itself from https://aiforscientists.eu/skills/job-babysitter/brief.md.

# Skill brief: job-babysitter

Source: https://aiforscientists.eu/skills/job-babysitter/
Recipe: Let checking cost set freedom (https://aiforscientists.eu/recipes/06-let-checking-cost-decide/)

Build a skill for my own setting from this brief.
If you have a skill-creation skill or tool, use it.
Before you write anything, ask me each question under "Local details", one at a time.
Do not guess an answer; if I do not know one, leave a marked TODO in the skill.
Keep every rule under "Never" in the skill you write.
Write the check under "How to check the output" into the skill, so that it runs every time.
Then run the skill once on the test case and show me the result.

## Purpose

Watches cluster jobs, resubmits failed runs up to a set limit, and logs each change.

## Inputs

- The list of jobs to watch.
- The code, configuration and log locations of each job.

## Output

- A log with one line per event: job, failure cause, fix applied, resubmission.

## Steps

1. Check the state of each job at a set interval.
2. For each failed job, read the log and name the cause.
3. If the cause is trivial and on the allowed list, fix it and resubmit.
4. Otherwise stop and report the job to the user.
5. Write every action to the log.

## Local details

1. Which scheduler does your cluster use (for example Slurm), and which commands may the agent run?
2. Which errors may the agent fix alone, for example out-of-memory or time limit?
3. What is the maximum number of resubmissions per job, and the quota the agent must stay under?
4. Which paths may the agent read and write?
5. How should it reach you when it stops?

## Never

- Never change the scientific code or the analysis parameters.
- Never delete files or checkpoints.
- Never exceed the resubmission limit or the group's quota.

## How to check the output

The log lists each resubmission and its cause. The code and parameters of each run are unchanged, which a diff confirms.

## Test case

Submit a job that fails with too little memory. The skill should raise the memory once, resubmit, and log it.

Local details¶

Your version needs answers to these questions:

Which scheduler does your cluster use (for example Slurm), and which commands may the agent run?
Which errors may the agent fix alone, for example out-of-memory or time limit?
What is the maximum number of resubmissions per job, and the quota the agent must stay under?
Which paths may the agent read and write?
How should it reach you when it stops?

Never¶

Never change the scientific code or the analysis parameters.
Never delete files or checkpoints.
Never exceed the resubmission limit or the group's quota.

Local versions¶

No local version is published yet. If you built one, share it as an example for others.

Discussion

Used this recipe or skill, or seen it fail? Say so here. Comments need a GitHub account; without one, write to aiforscience@googlegroups.com.

job-babysitter¶