# Skill brief: public-data-qc

Source: https://aiforscientists.eu/skills/public-data-qc/
Recipe: Pin down the method (https://aiforscientists.eu/recipes/08-pin-down-the-method/)

Build a skill for my own setting from this brief.
If you have a skill-creation skill or tool, use it.
Before you write anything, ask me each question under "Local details", one at a time.
Do not guess an answer; if I do not know one, leave a marked TODO in the skill.
Keep every rule under "Never" in the skill you write.
Write the check under "How to check the output" into the skill, so that it runs every time.
Then run the skill once on the test case and show me the result.

## Purpose

Downloads, deduplicates and checks public datasets, with a report for each step.

## Inputs

- The list of datasets with their accession numbers or URLs.
- The checks the data must pass.

## Output

- The data, stored in the agreed place.
- A report per dataset: source, checksum, counts before and after each step, duplicates removed.

## Steps

1. Download each dataset and record its checksum.
2. Check the counts against the source's metadata.
3. Find and remove duplicates, and list them.
4. Run the agreed checks.
5. Write the report.

## Local details

1. Which repositories do you download from?
2. Where may the data be stored, and how much space is there?
3. Which checks must each dataset pass?
4. Which data must not be downloaded, for example controlled-access data?

## Never

- Never download controlled-access data without the approval.
- Never change values in the data; only remove listed duplicates.

## How to check the output

Checksums and counts for each step can be compared with the source.

## Test case

Two public datasets that share some samples. The report should list the shared samples.