skillZs
★ LIVE SKILL TAGS ★
>>> LIVE SKILLS INDEX <<<
* OPEN SOURCE *
NO LOGIN, NO TRACKING
※ REAL INSTALL DATA ※
← back to all skills
owid/etl49 installs

update-dataset

End-to-end dataset update workflow with PR creation, snapshot, meadow, garden, and grapher steps. Use when user wants to update a dataset, refresh data, run ETL update, or mentions updating dataset versions.

How do I install this agent skill?

npx skills add https://github.com/owid/etl --skill update-dataset
view source ↗

Is this agent skill safe to install?

  • Gen Agent Trust Hubpass

    This skill manages complex dataset update workflows. It is generally safe but contains an indirect prompt injection surface because it processes external data and has the ability to modify code and execute database queries.

  • Socketwarn

    1 alert: gptAnomaly

  • Snykpass

    Risk: LOW · No issues

  • Runlayerfail

    1/2 files flagged

What does this agent skill do?

Update Dataset (PR → snapshot → steps → grapher)

Use this skill to run a complete dataset update with Claude Code subagents, keep a live progress checklist, and pause for approval at a checkpoint after every numbered workflow step before continuing.

Inputs

  • <namespace>/<old_version>/<name>
  • Get <new_version> as today's date by running date -u +"%Y-%m-%d"

Optional trailing args:

  • branch: The working branch name (defaults to current branch)

Assumptions:

  • All artifacts are written to workbench/<short_name>/.
  • Persist progress to workbench/<short_name>/progress.md and update it after each step.

Progress checklist (maintain, tick live, and persist to progress.md)

(Checkpoint rule: After you finish each item below that represents a workflow step, immediately run the CHECKPOINT procedure. Do not batch multiple steps before a checkpoint.)

  • Parse inputs and resolve: channel, namespace, version, short_name, old_version, branch
  • Clean workbench directory: delete workbench/<short_name> unless continuing existing update
  • Run ETL update workflow via etl-update subagent (help → dry run → approval → real run)
  • Create or reuse draft PR and work branch
  • Update snapshot and compare to previous version; capture summary
  • Meadow step: run + fix + diff + summarize
  • Garden step: run + fix + diff + summarize
  • Grapher step: run + verify (skip diffs), or explicitly mark N/A
  • CHECKPOINT — present consolidated summary and request approval
  • If approved, commit, push, and update PR description
  • Optional: run indicator upgrade on staging and persist report
  • Draft Slack announcement and notify user to post it to #data-updates-comms

Persistence:

  • After ticking each item, update workbench/<short_name>/progress.md with the current checklist state and a timestamp.

CHECKPOINT (mandatory user approval)

Always performed immediately after completing each numbered workflow step (1–6). Never start the next step until approval is granted.

Procedure (each time):

  1. Present a concise summary of what just changed, key diffs/issues resolved, and what the next step will do.
  2. Ask exactly: Proceed? reply: yes/no
  3. Only continue if the user replies exactly yes (case-insensitive). Any other reply = no; stop and wait.
  4. On approval:
    • Update progress checklist (tick the completed item) and write workbench/<short_name>/progress.md with timestamp.
    • Commit related changes (if any), push.
    • Update (or append to) the PR description: add a collapsed section titled with the step name (e.g., "Snapshot Update", "Meadow Update") containing the summary.

Mandatory per-step checkpoints (rule)

You MUST:

  • Stop after each workflow step (1–6) and run CHECKPOINT before starting the next (step 7 is optional and still requires a checkpoint if executed).
  • Never chain multiple steps inside a single approval.
  • Treat missing or ambiguous replies as no.

Workflow orchestration

  1. Initial setup

    • Check if workbench/<short_name>/progress.md exists to determine if continuing existing update
    • If starting fresh: delete workbench/<short_name> directory if it exists
    • Create fresh workbench/<short_name> directory for artifacts
  2. Run ETL update command (etl-update subagent)

    • Inputs: <namespace>/<old_version>/<short_name> plus any required flags
    • CRITICAL: Run etl update ONCE for the full step URI (e.g., data://garden/namespace/old_version/short_name). Do NOT run it separately per channel (snapshot, meadow, garden, grapher). Running it once ensures all cross-step DAG dependencies are updated together. Running it per-channel leaves stale version references in dag/main.yml (e.g., garden pointing to old meadow version).
    • Perform help check, dry run, approval, then real execution; capture summary for later PR notes
    • After running, always verify dag/main.yml: grep for the old version and confirm all internal references between the new steps point to the new version (e.g., garden depends on new meadow, not old meadow).
    • CHECKPOINT (stop → summarize → ask → require yes)
  3. Create PR and integrate update via subagent (etl-pr)

    • Inputs: <namespace>/<old_version>/<short_name>
    • Create or reuse draft PR, set up work branch, and incorporate the ETL update outputs
    • CHECKPOINT
  4. Snapshot run & compare (snapshot-runner subagent)

    • Inputs: <namespace>/<new_version>/<short_name> and <old_version>
    • CHECKPOINT
  5. Meadow step repair/verify (step-fixer subagent, channel=meadow)

    • Run, fix, re-run; produce diffs
    • Save diffs and summaries
    • CHECKPOINT
  6. Garden step repair/verify (step-fixer subagent, channel=garden)

    • Run, fix, re-run; produce diffs
    • Save diffs and summaries
    • CHECKPOINT
  7. Grapher step run/verify (step-fixer subagent, channel=grapher, add --grapher)

    • Skip diff
    • CHECKPOINT
  8. Indicator upgrade (optional, staging only)

    • Use indicator-upgrader subagent with <short_name> <branch>
    • CRITICAL: After the upgrader finishes, always verify it actually worked by querying staging: make query SQL="SELECT COUNT(*) FROM chart_dimensions cd JOIN variables v ON cd.variableId = v.id WHERE v.catalogPath LIKE '%<namespace>/<new_version>%'". If the count is 0, the upgrade did not run — re-run it.
    • CHECKPOINT (if executed)
  9. Slack announcement

    • Fill out the template at .claude/skills/update-dataset/slack-announcement-template.md using facts gathered during the update (coverage, chart count, key changes, etc.)
    • Ask user if unsure about any details
    • Save the draft to workbench/<short_name>/slack-announcement.md
    • Tell the user: "Slack announcement drafted at workbench/<short_name>/slack-announcement.md. Please review and post it to #data-updates-comms."

Guardrails and tips

  • DAG consistency: After etl update, always verify that all new steps in dag/main.yml reference each other with the new version. A common bug is garden depending on old meadow or old snapshot — this silently loads stale data.
  • Never return empty tables or comment out logic as a workaround — fix the parsing/transformations instead.
  • Column name changes: update garden processing code and metadata YAMLs (garden/grapher) to match schema changes.
  • Indexing: avoid leaking index columns from reset_index(); format tables with tb.format(["country", "year"]) as appropriate.
  • Metadata validation errors are guidance — update YAML to add/remove variables as indicated.

Artifacts (expected)

  • workbench/<short_name>/snapshot-runner.md
  • workbench/<short_name>/progress.md
  • workbench/<short_name>/meadow_diff_raw.txt and meadow_diff.md
  • workbench/<short_name>/garden_diff_raw.txt and garden_diff.md
  • workbench/<short_name>/indicator_upgrade.json (if indicator-upgrader was used)

Example usage

  • Minimal catalog URI with explicit old version:
    • update-dataset data://snapshot/irena/2024-11-15/renewable_power_generation_costs 2023-11-15 update-irena-costs

Common issues when data structure changes

  • SILENT FAILURES WARNING: Never return empty tables or comment code as workarounds!
  • Column name changes: If columns are renamed/split (e.g., single cost → local currency + PPP), update:
    • Python code references in the garden step
    • Garden metadata YAML (e.g., food_prices_for_nutrition.meta.yml)
    • Grapher metadata YAML (if it exists)
  • Index issues: Check for unwanted index columns from reset_index() — ensure proper indexing with tb.format(["country", "year"]).
  • Metadata validation: Use error messages as a guide — they show exactly which variables to add/remove from YAML files.

Add the canonical catalog link to the repository README so users can inspect current installs and available audits. The publishing guide covers the complete discovery path.

<a href="https://skillzs.dev/skills/owid/etl/update-dataset">View update-dataset on skillZs</a>