Skip to content

public-data-qc

Downloads, deduplicates and checks public datasets, with a report for each step.

  • Recipe: Pin down the method
  • Output check: Machine-checkable Checksums and counts for each step can be compared with the source.
  • Status: brief

Make your own version

This skill depends on local details, so the site publishes a brief instead of one institute's skill. Copy the brief and give it to your agent. If your agent has a skill-creation skill, such as Anthropic's skill-creator, it will use it. The agent asks you the questions under "Local details", builds the skill, and runs it once on the test case.

An agent that can read web pages can also fetch the brief itself from https://aiforscientists.eu/skills/public-data-qc/brief.md.

# Skill brief: public-data-qc

Source: https://aiforscientists.eu/skills/public-data-qc/
Recipe: Pin down the method (https://aiforscientists.eu/recipes/08-pin-down-the-method/)

Build a skill for my own setting from this brief.
If you have a skill-creation skill or tool, use it.
Before you write anything, ask me each question under "Local details", one at a time.
Do not guess an answer; if I do not know one, leave a marked TODO in the skill.
Keep every rule under "Never" in the skill you write.
Write the check under "How to check the output" into the skill, so that it runs every time.
Then run the skill once on the test case and show me the result.

## Purpose

Downloads, deduplicates and checks public datasets, with a report for each step.

## Inputs

- The list of datasets with their accession numbers or URLs.
- The checks the data must pass.

## Output

- The data, stored in the agreed place.
- A report per dataset: source, checksum, counts before and after each step, duplicates removed.

## Steps

1. Download each dataset and record its checksum.
2. Check the counts against the source's metadata.
3. Find and remove duplicates, and list them.
4. Run the agreed checks.
5. Write the report.

## Local details

1. Which repositories do you download from?
2. Where may the data be stored, and how much space is there?
3. Which checks must each dataset pass?
4. Which data must not be downloaded, for example controlled-access data?

## Never

- Never download controlled-access data without the approval.
- Never change values in the data; only remove listed duplicates.

## How to check the output

Checksums and counts for each step can be compared with the source.

## Test case

Two public datasets that share some samples. The report should list the shared samples.

Local details

Your version needs answers to these questions:

  1. Which repositories do you download from?
  2. Where may the data be stored, and how much space is there?
  3. Which checks must each dataset pass?
  4. Which data must not be downloaded, for example controlled-access data?

Never

  • Never download controlled-access data without the approval.
  • Never change values in the data; only remove listed duplicates.

Local versions

No local version is published yet. If you built one, share it as an example for others.

Discussion

Used this recipe or skill, or seen it fail? Say so here. Comments need a GitHub account; without one, write to aiforscience@googlegroups.com.