skillZs
★ LIVE SKILL TAGS ★
>>> LIVE SKILLS INDEX <<<
* OPEN SOURCE *
NO LOGIN, NO TRACKING
※ REAL INSTALL DATA ※
← back to all skills
posthog/skills75 installs

investigating-ci-failures

Investigates one CI failure to a verdict: whose fault it is, which commit caused it, who wrote that commit, and whether a fix has landed. Use for "who broke master", "why did this test fail in CI", "is this failure my PR or everyone", "is this test flaky or broken", "when did this failure start", "what is broken in CI right now", and "why did the merge queue kick my PR". Works from the engineering-analytics-broken-tests tool, the CI failure log tools, and two warehouse views: engineering_analytics_ci_failures and engineering_analytics_ci_job_history. Not for aggregate CI health, cost, or merge bottlenecks (use diagnosing-ci-and-merge-bottlenecks), and not for saved insights (use turning-engineering-analytics-into-insights).

How do I install this agent skill?

npx skills add https://github.com/posthog/skills --skill investigating-ci-failures
view source ↗

Is this agent skill safe to install?

  • Gen Agent Trust Hubpass

    The skill provides a structured framework for diagnosing CI failures using SQL queries and specialized logging tools. It focuses on identifying breaking commits and flaky tests within the vendor's engineering analytics environment without any detected security risks.

  • Socketpass

    No alerts

  • Snykwarn

    Risk: MEDIUM · 1 issue

What does this agent skill do?

Investigating CI failures

Take one failing test or one red run to a verdict: the PR's own bug, a default-branch break, or a flaky test. For a default-branch break, also name the culprit commit, its author, its PR, and the fix if one landed. You never need to re-run CI to answer.

Choosing a starting point

QuestionUse
What is broken across CI right now?engineering-analytics-broken-tests
Why did my PR's CI fail?engineering-analytics-ci-failure-logs with the PR number
Why did this run fail?engineering-analytics-run-failure-logs with run_id and ci_engine
Who broke master? When did this failure start?The two views, with the steps below
Is this test flaky?The failure's shape, then engineering-analytics-flaky-tests
Is this setup or infrastructure failure common?Job conclusions, query 7

Query the two views with the execute-sql tool:

  • engineering_analytics_ci_failures has one row per pytest FAILED <nodeid> line from the CI logs. fingerprint joins the test id to a normalized error, so it groups one failure across runs.
  • engineering_analytics_ci_job_history has one row per job attempt, with its conclusion and commit attribution: head_sha, commit_author_name, commit_message, and commit_pr_number.

The logs hold failures only. Passing jobs exist only in the job history, so every question about when a job turned red or green must use it.

Every numbered query is in references/investigation-queries.md.

Triage with the broken-tests tool

engineering-analytics-broken-tests groups the last 2 days of failures by fingerprint and returns the most urgent first. Each row's state is one of breaking_master, blocking_merge_queue, novel_burst, potentially_resolved, flaky, or pr_only. The tool's description defines each state. breaking_master_jobs lists the default-branch jobs whose latest run is red. Pass a row's latest_run_id and latest_ci_engine to engineering-analytics-run-failure-logs to read the failing lines. The list stops at 200 rows, so pass test or branch to ask about one test or one PR.

Reading a failure's shape

Fingerprint the failure with query 1, then read the branch count, the time window, and the master hits.

ShapeReadingNext step
One PR branchThat PR's own problemRead its failure lines.
One trunk-merge/pr-<n>/... gate branchA conflict with trunk or with a PR queued aheadRun query 8, then compare the gate branch with trunk.
Many branches in a tight window, with master hitsA default-branch breakRun query 2 to find the culprit.
Many branches, scattered over days or weeksFlakyConfirm with engineering-analytics-flaky-tests.

PR CI tests the PR merged with the default branch. One bad commit there fails every PR that runs at the same time. When the failure is not the asker's fault, say so in the first sentence.

A merge-queue gate branch carries trunk, the PR, and the PRs queued ahead of it. The PR already passed its own CI, so a failure there is not that PR's bug by default.

Finding the culprit of a default-branch break

Query 2 lists the failing job's history on the default branch in time order. The first failure after a success is the culprit row. The first success after the red streak is the fix row. Each row carries the SHA, the author, the commit message, and commit_pr_number.

Check two things before you name a commit:

  • The commit's message or diff should plausibly touch the failing test's area. If it does not, widen the window and check the adjacent commits.
  • The fingerprint's first_seen should fall just after the culprit merged. A mismatch means two different problems share one test.

A revert appears as a new commit whose commit_pr_number is the reverting PR.

Checking for a flake

Read the other attempts before you say "flaky". run_attempt > 1 rows are re-runs of the same job on the same commit. A failure that clears on a later attempt suggests a flake, and query 6 tracks the fingerprint across attempts. A branch where every run is red has a deterministic failure, and a re-run changes none of its inputs.

engineering-analytics-flaky-tests marks a test confirmed_flake only when one commit both failed and passed it. Its counts are absolute and never rates, because CI does not emit fast passing runs.

Caveats

  • Fingerprints cover pytest only. For Jest, Playwright, or Cargo failures, read the raw lines with the two failure-log tools.
  • A job that failed before its tests ran writes no FAILED line. No test-level tool sees it. Only job conclusions show it, through query 7.
  • The logs arrive within minutes, but the job history arrives by webhook sync and can lag. During a live incident, run query 5 before you trust a boundary.
  • Keep ci_engine with every run_id. A null engine means unknown provenance, so do not join it to a known engine by integer id.
  • Put a time bound on every query. Logs retention is short.
  • For the job history, add a string floor such as created_at_raw >= '<YYYY-MM-DD>', one day below the window. The parsed created_at filter alone forces a full scan.
  • On depot-* runners, actions/cache reads Depot Cache, which the GitHub cache API cannot see. Read the job's Cache not found for input keys: line before you blame a cache eviction.

Reporting

Lead with the verdict in plain words, including whether the asker's PR is at fault. Then give the evidence: the boundary rows, the fingerprint window, and the occurrence and branch counts. Name the author as a fact, not as blame, and link the commit and the PR.

Add the canonical catalog link to the repository README so users can inspect current installs and available audits. The publishing guide covers the complete discovery path.

<a href="https://skillzs.dev/skills/posthog/skills/investigating-ci-failures">View investigating-ci-failures on skillZs</a>