investigating-ci-failures
Investigates one CI failure to a verdict: whose fault it is, which commit caused it, who wrote that commit, and whether a fix has landed. Use for "who broke master", "why did this test fail in CI", "is this failure my PR or everyone", "is this test flaky or broken", "when did this failure start", "what is broken in CI right now", and "why did the merge queue kick my PR". Works from the engineering-analytics-broken-tests tool, the CI failure log tools, and two warehouse views: engineering_analytics_ci_failures and engineering_analytics_ci_job_history. Not for aggregate CI health, cost, or merge bottlenecks (use diagnosing-ci-and-merge-bottlenecks), and not for saved insights (use turning-engineering-analytics-into-insights).
How do I install this agent skill?
npx skills add https://github.com/posthog/skills --skill investigating-ci-failuresIs this agent skill safe to install?
- Gen Agent Trust Hubpass
The skill provides a structured framework for diagnosing CI failures using SQL queries and specialized logging tools. It focuses on identifying breaking commits and flaky tests within the vendor's engineering analytics environment without any detected security risks.
- Socketpass
No alerts
- Snykwarn
Risk: MEDIUM · 1 issue
What does this agent skill do?
Investigating CI failures
Take one failing test or one red run to a verdict: the PR's own bug, a default-branch break, or a flaky test. For a default-branch break, also name the culprit commit, its author, its PR, and the fix if one landed. You never need to re-run CI to answer.
Choosing a starting point
| Question | Use |
|---|---|
| What is broken across CI right now? | engineering-analytics-broken-tests |
| Why did my PR's CI fail? | engineering-analytics-ci-failure-logs with the PR number |
| Why did this run fail? | engineering-analytics-run-failure-logs with run_id and ci_engine |
| Who broke master? When did this failure start? | The two views, with the steps below |
| Is this test flaky? | The failure's shape, then engineering-analytics-flaky-tests |
| Is this setup or infrastructure failure common? | Job conclusions, query 7 |
Query the two views with the execute-sql tool:
engineering_analytics_ci_failureshas one row per pytestFAILED <nodeid>line from the CI logs.fingerprintjoins the test id to a normalized error, so it groups one failure across runs.engineering_analytics_ci_job_historyhas one row per job attempt, with itsconclusionand commit attribution:head_sha,commit_author_name,commit_message, andcommit_pr_number.
The logs hold failures only. Passing jobs exist only in the job history, so every question about when a job turned red or green must use it.
Every numbered query is in references/investigation-queries.md.
Triage with the broken-tests tool
engineering-analytics-broken-tests groups the last 2 days of failures by fingerprint and returns the most urgent first.
Each row's state is one of breaking_master, blocking_merge_queue, novel_burst, potentially_resolved, flaky, or pr_only.
The tool's description defines each state.
breaking_master_jobs lists the default-branch jobs whose latest run is red.
Pass a row's latest_run_id and latest_ci_engine to engineering-analytics-run-failure-logs to read the failing lines.
The list stops at 200 rows, so pass test or branch to ask about one test or one PR.
Reading a failure's shape
Fingerprint the failure with query 1, then read the branch count, the time window, and the master hits.
| Shape | Reading | Next step |
|---|---|---|
| One PR branch | That PR's own problem | Read its failure lines. |
One trunk-merge/pr-<n>/... gate branch | A conflict with trunk or with a PR queued ahead | Run query 8, then compare the gate branch with trunk. |
| Many branches in a tight window, with master hits | A default-branch break | Run query 2 to find the culprit. |
| Many branches, scattered over days or weeks | Flaky | Confirm with engineering-analytics-flaky-tests. |
PR CI tests the PR merged with the default branch. One bad commit there fails every PR that runs at the same time. When the failure is not the asker's fault, say so in the first sentence.
A merge-queue gate branch carries trunk, the PR, and the PRs queued ahead of it. The PR already passed its own CI, so a failure there is not that PR's bug by default.
Finding the culprit of a default-branch break
Query 2 lists the failing job's history on the default branch in time order.
The first failure after a success is the culprit row.
The first success after the red streak is the fix row.
Each row carries the SHA, the author, the commit message, and commit_pr_number.
Check two things before you name a commit:
- The commit's message or diff should plausibly touch the failing test's area. If it does not, widen the window and check the adjacent commits.
- The fingerprint's
first_seenshould fall just after the culprit merged. A mismatch means two different problems share one test.
A revert appears as a new commit whose commit_pr_number is the reverting PR.
Checking for a flake
Read the other attempts before you say "flaky".
run_attempt > 1 rows are re-runs of the same job on the same commit.
A failure that clears on a later attempt suggests a flake, and query 6 tracks the fingerprint across attempts.
A branch where every run is red has a deterministic failure, and a re-run changes none of its inputs.
engineering-analytics-flaky-tests marks a test confirmed_flake only when one commit both failed and passed it.
Its counts are absolute and never rates, because CI does not emit fast passing runs.
Caveats
- Fingerprints cover pytest only. For Jest, Playwright, or Cargo failures, read the raw lines with the two failure-log tools.
- A job that failed before its tests ran writes no
FAILEDline. No test-level tool sees it. Only job conclusions show it, through query 7. - The logs arrive within minutes, but the job history arrives by webhook sync and can lag. During a live incident, run query 5 before you trust a boundary.
- Keep
ci_enginewith everyrun_id. A null engine means unknown provenance, so do not join it to a known engine by integer id. - Put a time bound on every query. Logs retention is short.
- For the job history, add a string floor such as
created_at_raw >= '<YYYY-MM-DD>', one day below the window. The parsedcreated_atfilter alone forces a full scan. - On
depot-*runners,actions/cachereads Depot Cache, which the GitHub cache API cannot see. Read the job'sCache not found for input keys:line before you blame a cache eviction.
Reporting
Lead with the verdict in plain words, including whether the asker's PR is at fault. Then give the evidence: the boundary rows, the fingerprint window, and the occurrence and branch counts. Name the author as a fact, not as blame, and link the commit and the PR.
How can the creator link this skill?
Add the canonical catalog link to the repository README so users can inspect current installs and available audits. The publishing guide covers the complete discovery path.
<a href="https://skillzs.dev/skills/posthog/skills/investigating-ci-failures">View investigating-ci-failures on skillZs</a>