# Skill brief: job-babysitter

Source: https://aiforscientists.eu/skills/job-babysitter/
Recipe: Let checking cost set freedom (https://aiforscientists.eu/recipes/06-let-checking-cost-decide/)

Build a skill for my own setting from this brief.
If you have a skill-creation skill or tool, use it.
Before you write anything, ask me each question under "Local details", one at a time.
Do not guess an answer; if I do not know one, leave a marked TODO in the skill.
Keep every rule under "Never" in the skill you write.
Write the check under "How to check the output" into the skill, so that it runs every time.
Then run the skill once on the test case and show me the result.

## Purpose

Watches cluster jobs, resubmits failed runs up to a set limit, and logs each change.

## Inputs

- The list of jobs to watch.
- The code, configuration and log locations of each job.

## Output

- A log with one line per event: job, failure cause, fix applied, resubmission.

## Steps

1. Check the state of each job at a set interval.
2. For each failed job, read the log and name the cause.
3. If the cause is trivial and on the allowed list, fix it and resubmit.
4. Otherwise stop and report the job to the user.
5. Write every action to the log.

## Local details

1. Which scheduler does your cluster use (for example Slurm), and which commands may the agent run?
2. Which errors may the agent fix alone, for example out-of-memory or time limit?
3. What is the maximum number of resubmissions per job, and the quota the agent must stay under?
4. Which paths may the agent read and write?
5. How should it reach you when it stops?

## Never

- Never change the scientific code or the analysis parameters.
- Never delete files or checkpoints.
- Never exceed the resubmission limit or the group's quota.

## How to check the output

The log lists each resubmission and its cause. The code and parameters of each run are unchanged, which a diff confirms.

## Test case

Submit a job that fails with too little memory. The skill should raise the memory once, resubmit, and log it.