exploring-llm-clusters
Investigate AI observability clusters — understand usage patterns in AI/LLM traffic, compare cluster behavior, compute cost/latency metrics, and drill into individual traces within clusters.
How do I install this agent skill?
npx skills add https://github.com/posthog/ai-plugin --skill exploring-llm-clustersIs this agent skill safe to install?
- Gen Agent Trust Hubpass
The skill is a legitimate PostHog extension for AI observability, enabling users to analyze clusters of LLM traffic. It uses standard SQL queries and a local helper script to process data. No malicious behavior or security vulnerabilities were identified.
- Socketpass
No alerts
- Snykpass
Risk: LOW · No issues
What does this agent skill do?
Exploring LLM clusters
Use this skill when investigating AI observability clusters — understanding what patterns exist in your AI/LLM traffic, comparing cluster behavior, and drilling into individual clusters.
Tools
| Tool | Purpose |
|---|---|
posthog:llma-clustering-job-list | List clustering job configurations for the team |
posthog:llma-clustering-job-get | Get a specific clustering job by ID |
posthog:execute-sql | Query cluster run events and compute metrics |
posthog:query-llm-traces-list | Find traces belonging to a cluster |
posthog:query-llm-trace | Inspect a specific trace in detail |
How clustering works
PostHog clusters LLM traces, individual generations, or evaluation events by embedding similarity.
A Temporal workflow runs periodically or on-demand, producing cluster events stored as
$ai_trace_clusters (trace-level), $ai_generation_clusters (generation-level), or
$ai_evaluation_clusters (evaluation-level).
Each cluster event contains:
$ai_clustering_run_id— unique run identifier (format:<team_id>_<level>_<YYYYMMDD>_<HHMMSS>[_<job_id>])$ai_clustering_level—"trace","generation", or"evaluation"$ai_window_start/$ai_window_end— time window of the data that was analyzed$ai_total_items_analyzed— number of traces, generations, or evaluations processed$ai_clusters— JSON array of cluster objects$ai_clustering_params— algorithm parameters used
The analyzed window closes when a run starts, and the cluster event lands once the run finishes.
So the cluster event's own timestamp is always after $ai_window_end, by anything from seconds to hours.
Use the window only to bound the traces, generations, and evaluations that were analyzed.
To find the cluster event itself, filter on $ai_clustering_run_id with a plain recent-time bound.
Cluster object shape (inside $ai_clusters)
{
"cluster_id": 0,
"size": 42,
"title": "User authentication flows",
"description": "Traces involving login, signup, and token refresh operations",
"traces": {
"<trace_or_generation_id>": {
"distance_to_centroid": 0.123,
"rank": 0,
"x": -2.34,
"y": 1.56,
"timestamp": "2026-03-28T10:00:00Z",
"trace_id": "abc-123",
"generation_id": "gen-456"
}
},
"centroid_x": -2.1,
"centroid_y": 1.4
}
cluster_id: -1is the noise/outlier cluster (items that didn't fit any cluster)- Items in
tracesare keyed by trace ID (trace-level), generation event UUID (generation-level), or evaluation event UUID (evaluation-level) rankorders items by proximity to centroid (0 = closest)x,yare 2D coordinates for visualization (UMAP/PCA/t-SNE reduced)
Clustering jobs
Each team can have up to 10 clustering jobs. A job defines:
- name — human-readable label
- analysis_level —
"trace","generation", or"evaluation" - event_filters — property filters scoping which items are included
- enabled — whether the job runs on schedule
Default jobs named "Default - traces", "Default - generations", and "Default - evaluations" are auto-created
and disabled when a custom job is created for the same level.
Workflow: explore clusters
Step 1 — List recent clustering runs
posthog:execute-sql
SELECT
toString(properties.$ai_clustering_run_id) AS run_id,
toString(properties.$ai_clustering_level) AS level,
toString(properties.$ai_clustering_job_id) AS job_id,
toString(properties.$ai_clustering_job_name) AS job_name,
toString(properties.$ai_window_start) AS window_start,
toString(properties.$ai_window_end) AS window_end,
toFloat64OrNull(toString(properties.$ai_total_items_analyzed)) AS total_items,
timestamp
FROM events
WHERE event IN ('$ai_trace_clusters', '$ai_generation_clusters', '$ai_evaluation_clusters')
AND timestamp >= now() - INTERVAL 14 DAY
ORDER BY timestamp DESC
LIMIT 10
Step 2 — Get clusters from a specific run
posthog:execute-sql
SELECT
toString(properties.$ai_clustering_run_id) AS run_id,
toString(properties.$ai_clustering_level) AS level,
toString(properties.$ai_clustering_job_id) AS job_id,
toString(properties.$ai_clustering_job_name) AS job_name,
toString(properties.$ai_window_start) AS window_start,
toString(properties.$ai_window_end) AS window_end,
toFloat64OrNull(toString(properties.$ai_total_items_analyzed)) AS total_items,
properties.$ai_clusters AS clusters,
properties.$ai_clustering_params AS params,
timestamp
FROM events
WHERE event IN ('$ai_trace_clusters', '$ai_generation_clusters', '$ai_evaluation_clusters')
AND timestamp >= now() - INTERVAL 14 DAY
AND toString(properties.$ai_clustering_run_id) = '<run_id>'
ORDER BY timestamp DESC
LIMIT 1
Keep the lookback bound wide enough to cover the timestamp Step 1 reported for the run.
Never bound this query with $ai_window_start / $ai_window_end.
The cluster event is emitted after the window closes, so those bounds return zero rows.
The clusters field is a JSON array. Parse it to see cluster titles, sizes, descriptions, optional metrics, and each cluster's traces map.
Important: The clusters JSON can be very large (thousands of trace, generation, or evaluation IDs with coordinates).
When the result is too large for inline display, it auto-persists to a file.
Use print_clusters.py from scripts/ to get a readable summary.
Step 3 — Compute metrics for clusters
For trace-level clusters, compute cost/latency/token metrics.
The latency rule is the same as in query-llm-trace and query-llm-traces-list, so the numbers match the trace view:
posthog:execute-sql
SELECT
properties.$ai_trace_id AS trace_id,
round(sumIf(toFloat(properties.$ai_total_cost_usd),
event IN ('$ai_generation', '$ai_embedding')), 10) AS total_cost,
round(
coalesce(
-- The root $ai_trace event reports the wall-clock latency of the whole trace.
nullIf(maxIf(toFloat(properties.$ai_latency),
event = '$ai_trace' AND toFloat(properties.$ai_latency) > 0), 0),
CASE
WHEN countIf(toFloat(properties.$ai_latency) > 0 AND event != '$ai_generation') = 0
AND countIf(toFloat(properties.$ai_latency) > 0 AND event = '$ai_generation') > 0
THEN sumIf(toFloat(properties.$ai_latency),
event = '$ai_generation' AND toFloat(properties.$ai_latency) > 0)
ELSE sumIf(toFloat(properties.$ai_latency),
properties.$ai_parent_id IS NULL
OR toString(properties.$ai_parent_id) = toString(properties.$ai_trace_id))
END
), 2
) AS latency,
countIf(event = '$ai_trace' AND toFloat(properties.$ai_latency) > 0) > 0 AS has_root_latency,
sumIf(toInt(properties.$ai_input_tokens),
event IN ('$ai_generation', '$ai_embedding')) AS input_tokens,
sumIf(toInt(properties.$ai_output_tokens),
event IN ('$ai_generation', '$ai_embedding')) AS output_tokens,
countIf(isNotNull(properties.$ai_error) OR properties.$ai_is_error = 'true') AS error_count
FROM events
WHERE event IN ('$ai_trace', '$ai_generation', '$ai_embedding', '$ai_span')
AND timestamp >= parseDateTimeBestEffort('<window_start>')
AND timestamp <= parseDateTimeBestEffort('<window_end>')
AND properties.$ai_trace_id IN ('<trace_id_1>', '<trace_id_2>', ...)
GROUP BY trace_id
How latency is resolved, in order:
- A positive
$ai_latencyon the root$ai_traceevent. This is the measured wall-clock duration of the trace. - If no root latency exists and only generations carry latency, the sum of the generation latencies.
- Otherwise, the sum of the latencies on the trace's direct children (events with no
$ai_parent_id, or a parent equal to the trace ID).
Steps 2 and 3 are approximations.
They overstate the duration when children ran in parallel, and they count a nested event that has no $ai_parent_id as a direct child.
Use has_root_latency to tell measured durations from approximations, and mention approximations when you compare clusters.
Do not use max($ai_latency) across child events as the trace latency.
It returns the longest single step, not the trace duration.
For example, a trace runs three root-level spans one after another: 12 s, 9 s, and 6 s, with some time between them.
Its $ai_trace event reports 30 s.
max returns 12 s and the recipe above returns 30 s.
Cluster averages built from max understate latency and can put the clusters in the wrong order.
Cost and tokens only sum $ai_generation and $ai_embedding events, the same as the trace view.
The query reads $ai_trace and $ai_span events for latency only, so their properties cannot count the same spend twice.
The trace window bounds can cut off the root $ai_trace event, because it is usually the last event of the trace.
If has_root_latency is false for traces that should have one, widen <window_end> a little.
For generation-level clusters, match by event UUID:
posthog:execute-sql
SELECT
toString(uuid) as generation_id,
toFloat(properties.$ai_total_cost_usd) as cost,
toFloat(properties.$ai_latency) as latency,
toInt(properties.$ai_input_tokens) as input_tokens,
toInt(properties.$ai_output_tokens) as output_tokens,
if(properties.$ai_is_error = 'true', 1, 0) as is_error
FROM events
WHERE event = '$ai_generation'
AND timestamp >= parseDateTimeBestEffort('<window_start>')
AND timestamp <= parseDateTimeBestEffort('<window_end>')
AND uuid IN ('<gen_uuid_1>', '<gen_uuid_2>', ...)
For evaluation-level clusters, first check each cluster's metrics field from $ai_clusters (for example pass rate, N/A rate, dominant evaluator name, and average judge cost). When you need individual evaluation rows, match by event UUID:
posthog:execute-sql
SELECT
toString(uuid) AS evaluation_id,
toString(properties.$ai_trace_id) AS trace_id,
toString(properties.$ai_target_event_id) AS generation_id,
toString(properties.$ai_evaluation_name) AS evaluation_name,
toString(properties.$ai_evaluation_result) AS evaluation_result,
toString(properties.$ai_evaluation_reasoning) AS evaluation_reasoning,
toFloatOrNull(toString(properties.$ai_total_cost_usd)) AS judge_cost,
timestamp
FROM events
WHERE event = '$ai_evaluation'
AND timestamp >= parseDateTimeBestEffort('<window_start>')
AND timestamp <= parseDateTimeBestEffort('<window_end>')
AND uuid IN ('<eval_uuid_1>', '<eval_uuid_2>', ...)
Step 4 — Drill into specific traces
Once you've identified interesting clusters, use the trace tools to inspect individual traces:
posthog:query-llm-trace
{
"traceId": "<trace_id_from_cluster>",
"dateRange": {"date_from": "<window_start>", "date_to": "<window_end>"}
}
When you need message content
Use events for cluster events, IDs, cost/latency/token metrics, and evaluation rows.
Do not query events.properties.$ai_input, $ai_output, or $ai_output_choices when you need user messages or full model inputs/outputs —
those heavy fields live on posthog.ai_events.
For a few representative examples, prefer query-llm-trace; it reads posthog.ai_events for you and returns the full event tree.
For batch extraction, first get the trace IDs from the cluster, then query posthog.ai_events anchored on trace_id:
posthog:execute-sql
SELECT
trace_id,
timestamp,
span_id,
event,
model,
input,
output_choices
FROM posthog.ai_events
WHERE trace_id IN ('<trace_id_1>', '<trace_id_2>', ...)
ORDER BY trace_id, timestamp
posthog.ai_events has a shorter retention window than events; older clusters may still have metadata and metrics but no message content.
For the full $ai_* event schema, load the exploring-llm-traces skill.
Investigation patterns
"What kinds of LLM usage do we have?"
- List recent clustering runs (Step 1)
- Load the latest run's clusters (Step 2)
- Review cluster titles and descriptions — each represents a distinct usage pattern
- Compare cluster sizes to understand traffic distribution
"Which cluster is most expensive / slowest?"
- Load clusters from a run (Step 2)
- Extract trace IDs from each cluster
- Compute metrics per cluster (Step 3)
- Aggregate:
avg(cost),avg(latency),sum(cost)per cluster - Compare across clusters
"What's in this cluster?"
- Load the cluster's traces (from the
tracesfield) - Sort by
rank(closest to centroid = most representative) - Inspect the top 3-5 traces via
query-llm-traceto understand the pattern - Check the cluster
titleanddescriptionfor the AI-generated summary
"Are there error-heavy clusters?"
- Compute metrics (Step 3) with
error_count - Calculate error rate per cluster:
items_with_errors / total_items - Focus on clusters with high error rates
- Drill into errored traces to find root causes
- If a cluster shows a failure that keeps recurring, offer to catch it with an online evaluation scoped to that cluster (
creating-online-evaluations)
"How do clusters compare across runs?"
- List multiple runs (Step 1)
- Load clusters from each run
- Compare cluster titles — similar titles across runs indicate stable patterns
- Track cluster size changes to detect shifts in traffic patterns
Constructing UI links
- Clusters overview:
https://app.posthog.com/ai-observability/clusters - Specific run:
https://app.posthog.com/ai-observability/clusters/<url_encoded_run_id> - Cluster detail:
https://app.posthog.com/ai-observability/clusters/<url_encoded_run_id>/<cluster_id>
Always surface these links so the user can verify visually in the PostHog UI.
Tips
- Always set a time range in SQL queries — cluster events without time bounds are slow
- Bound a search for a cluster event by when it was emitted, not by the window it analyzed — a run's
$ai_window_endis earlier than the event's owntimestamp - Start with run listing to orient, then drill into specific clusters
- Cluster titles and descriptions are AI-generated summaries — verify by inspecting traces
- The noise cluster (
cluster_id: -1) contains outliers that didn't fit any pattern - Use
llma-clustering-job-listto understand what clustering configs are active - Trace IDs in clusters can be used directly with
query-llm-tracefor deep inspection - Message content lives on
posthog.ai_events, notevents.properties; usequery-llm-traceunless you need custom batch SQL - For large clusters, inspect the top-ranked traces (closest to centroid) for representative examples
How can the creator link this skill?
Add the canonical catalog link to the repository README so users can inspect current installs and available audits. The publishing guide covers the complete discovery path.
<a href="https://skillzs.dev/skills/posthog/ai-plugin/exploring-llm-clusters">View exploring-llm-clusters on skillZs</a>