kubesense-metrics

KubeSense Metrics

Metrics are queried with PromQL plus the MetricsQL extensions (see below). Requires the KubeSense MCP server; see kubesense-mcp for connection and auth.

Tools

Tool	Purpose	Notes
get-available-metrics	List metric names	search_keywords are AND-matched, case-insensitive substrings
get-metric-labels	List label names on one metric	name required
analyze-metrics	Execute PromQL	from_time, to_time, query_type, promql all required

All three resolve RBAC against the infrastructure module — a permission error here means the caller lacks infrastructure access, not that the metric is missing.

Discovery-First Rule

Never guess a metric name. Metric names are deployment-specific: they depend on which collectors a cluster ships (kube-state-metrics vs OTel, node-exporter vs hostmetrics). A query against a metric that does not exist returns an empty result that is indistinguishable from "the value is zero".

get-available-metrics  →  get-metric-labels  →  analyze-metrics

get-available-metrics defaults to the last 1 hour. A sparse metric that reported 90 minutes ago will not appear — widen from_time/to_time before concluding it does not exist.

{ "search_keywords": ["container", "memory"], "limit": 40 }

Both keywords must appear in the name, so ["container","memory"] matches container_memory_working_set_bytes but not container_cpu_usage_seconds_total.

get-metric-labels returns label names only, not values. To see values, query the metric with analyze-metrics and read the series labels off the result.

Label Conventions (the #1 cause of empty results)

[!IMPORTANT] The cluster label is clusterId — camelCase, no underscore. Not cluster, not kubesense_cluster, not cluster_id. Every kube-state-metrics / cAdvisor series in KubeSense carries it.

Concept	Label on k8s/cAdvisor metrics	Label on OTel hostmetrics
cluster	clusterId	kubesense_cluster
host / node	node	host_name
environment	—	kubesense_env_type
namespace	namespace	—
pod	pod	—
container	container	—
workload	deployment / daemonset / statefulset (per metric family)	—

The two families exist because a cluster ships telemetry either via node-exporter/process-exporter or via the OpenTelemetry collector. Run get-metric-labels to see which convention a given metric follows rather than assuming.

Cloud-provider metrics (AWS/GCP/Azure/Atlas/Confluent/Kong) are a third convention with none of these labels — see Cloud Resource Metrics below.

Process metrics: the exporter family uses groupname (formatted {pid}-{command}); the OTel family splits it into process_pid, process_command_line, and process_executable_name.

Writing Queries

{
  "from_time": "2026-07-30T10:00:00Z",
  "to_time": "2026-07-30T11:00:00Z",
  "query_type": "range",
  "promql": "sum(rate(container_cpu_usage_seconds_total{container!='',clusterId='prod-us'}[5m])) by (namespace)"
}

• query_type: "range" → a time series (trends). "instant" → one scalar per series (totals, current state, top-N).
• Step is computed for you: max(window/30, 15) seconds. You cannot set it.

Interval macros

A rate() window narrower than the step leaves gaps — the step for a 1-hour window is 120s, so rate(x[30s]) samples less than one bucket's worth and returns a broken series. Rather than picking a literal that happens to clear the step, use a macro. The API resolves these before the query reaches the metrics store:

Macro	Resolves to	Use for
$__rate_interval	max(step + 15s, 60s)	rate() and increase() — the default choice
$__interval	the step itself	*_over_time rollups, where the window should equal one bucket
$__time_filter	the query's whole time range (to - from)	a single total across the window

sum(rate(container_cpu_usage_seconds_total{container!='',clusterId='prod-us'}[$__rate_interval])) by (namespace)

$__rate_interval tracks the step at every window size, so the same query works over 5 minutes and over 7 days. A literal like [5m] is still valid and still a safe default, but it is the thing the macro exists to stop you having to reason about.

Two limits worth knowing:

• Only these three are resolved. $__interval_ms and $__range are not implemented and will reach the metrics store unexpanded, which fails.
• A macro inside a label value is left alone — up{job="$__interval"} matches the literal string, it is not rewritten into a duration.

Two container-metric rules

1. Always filter container!='' on container_* metrics. cAdvisor emits an extra pod-level rollup series with an empty container label; without the filter every sum() double-counts.
2. CPU is in cores. rate(container_cpu_usage_seconds_total[5m]) yields cores; multiply by 1000 for millicores, which is how KubeSense's own UI reports it.

kube_pod_container_resource_limits / _requests

These are a single metric for both CPU and memory, discriminated by a resource label — not separate metrics. Select it:

kube_pod_container_resource_limits{resource="memory"}
kube_pod_container_resource_limits{resource="cpu"}

Memory-usage-vs-limit is therefore a join, not a subtraction of two metrics:

sum by (pod, namespace) (container_memory_working_set_bytes{container!=''})
  / sum by (pod, namespace) (kube_pod_container_resource_limits{resource="memory"}) * 100

kube_*_status_phase / _condition are label-encoded

kube_pod_status_phase is 1/0 per phase, with the phase in a label — so counting running pods means selecting the label and de-duplicating series:

count(max by (pod, namespace, clusterId) (kube_pod_status_phase{phase="Running"}) > 0)

Same shape for nodes: kube_node_status_condition{condition="Ready",status="true"}. The max by (...) de-dup matters — duplicate label sets otherwise inflate the count.

MetricsQL Extensions

Queries run as MetricsQL, a superset of PromQL. The extensions you will reach for most:

Extension	Use
default 0	Substitute a value where the series is absent: sum(...) default 0
drop_empty_series(...)	Discard series that are entirely empty after filtering
[1h:60s] subqueries	Rollup-over-rollup, e.g. max_over_time(rate(x[5m])[1h:60s])
limit N	Cap an aggregate's output series: sum(x) by (pod) limit 10
keep_metric_names	Keep metric names through a function; fixes duplicate time series

These will not work against a stock Prometheus, but they are correct here.

MetricsQL reference: before using a function not shown in this file, check its signature and argument order, or reach for syntax beyond PromQL (WITH templates, or inside a selector, offset/@ placement, if/ifnot), read references/metricsql.md. It lists all 232 functions, each with what it does and whether stock PromQL has it.

Common Patterns

Goal	Query
Per-second counter rate	rate(metric[$__rate_interval])
Aggregate across series	sum(...) by (namespace)
Histogram percentile	histogram_quantile(0.99, sum(rate(metric_bucket[5m])) by (le))
Total over a window	increase(metric[$__time_filter])
Top N	topk(10, ...) — pair with query_type: "instant"
Regex label match	metric{namespace=~"prod-.*"}
Ratio as a percentage	sum(a) / sum(b) * 100

For a histogram percentile, always sum(rate(...)) by (le) before histogram_quantile — passing raw buckets grouped by other labels produces nonsense.

Cloud Resource Metrics

Metrics pulled from cloud providers' own monitoring APIs — AWS CloudWatch, Google Cloud Monitoring, Azure Monitor, MongoDB Atlas, Confluent Cloud, Kong — are in the same metrics store but follow a different convention entirely.

[!IMPORTANT] Every cloud datapoint, for every provider and resource type, is stored under one series name: kubesense_cloud_resource_metric. The provider's own metric name is the metric_name label, not part of the series name. get-available-metrics therefore returns exactly one name no matter how many cloud series exist, and searching it for CPUUtilization finds nothing.

kubesense_cloud_resource_metric{provider="aws",resource_type="Ec2Instance",metric_name="CPUUtilization"}

Fixed labels: provider (aws | gcp | azure | mongodbatlas | confluent | kong), account_id, resource_id, resource_type, metric_name, unit — plus collector dimensions such as region. There is no clusterId, namespace or pod, so a selector copied from a Kubernetes query matches nothing.

Start with list-cloud-integrations: it lists every connected account with its provider, account_id, status, and the resource_types reporting in the last hour. An empty cloud query is usually explained there (account paused, in error, or nothing reporting). Then discover metric names with PromQL, because those live in a label:

count by (resource_type) (kubesense_cloud_resource_metric{provider="aws"})
count by (metric_name, unit) (kubesense_cloud_resource_metric{resource_type="RdsInstance"})

Three rules that decide whether a cloud query is right or silently wrong:

1. Never rate() them. The provider's aggregation (CloudWatch Sum/Average, the Cloud Monitoring aligner, the Azure Monitor aggregation) is already applied at collection time. Use the raw value, or sum_over_time/avg_over_time/max_over_time to roll up buckets.
2. Always pin resource_type alongside metric_name. CPUUtilization alone mixes EC2, RDS, ECS, ElastiCache, DocumentDB, Neptune, OpenSearch and Redshift into one aggregate.
3. Collection is a 5-minute tick with a 15-minute forward-fill. Instant queries work because the newest sample is re-stamped; a series silent for more than 15 minutes is genuinely absent, not zero. Keep windows at or above 10m.

Full label contract, per-provider resource types and their metrics, the AWS types that roll up onto a parent (there is no PerformanceInsights or NetworkLoadBalancer resource type), and worked queries: references/cloud-metric-catalog.md.

Choosing Metrics vs Traces vs Infra Tools

• Resource pressure, saturation, capacity (CPU, memory, disk, GPU) → metrics.
• Request rate, latency percentiles, error rate per service/endpoint → traces (analyze-traces). Metrics have no per-request detail.
• "Is this pod restarting / why" or "what's running" → kubesense-infra tools; they return the same facts already joined and labelled, in one call.

Use metrics to quantify and confirm a hypothesis the cheaper tools surfaced.

Metric Catalog

For the metric families KubeSense collects — exact names for pods, containers, nodes, workloads, PVCs, network, GPU, JVM, and process metrics — read references/metric-catalog.md. For AWS, GCP, Azure, MongoDB Atlas, Confluent Cloud and Kong resource metrics, read references/cloud-metric-catalog.md.

Treat the catalog as what to expect, not a substitute for discovery: which families are present depends on the cluster's collectors, and application metrics are entirely deployment-specific. Always confirm with get-available-metrics.

Rules

1. get-available-metrics before writing PromQL. Never invent a metric name.
2. get-metric-labels before writing a label selector — and remember the cluster label is clusterId.
3. Filter container!='' on every container_* metric.
4. rate() windows must be ≥ the auto-computed step (max(window/30, 15)s). Prefer $__rate_interval, which guarantees it at any window size; [5m] is a safe literal.
5. kube_pod_container_resource_limits/_requests need a resource="cpu"|"memory" selector — they are one metric, not two.
6. kube_*_status_phase/_condition encode the state in a label; select it and de-dup with max by (...) before counting.
7. query_type: "instant" for totals/current state/top-N; "range" for trends.
8. An empty result means "no matching series", which is NOT the same as zero — verify the metric exists and the labels match before reporting a value of 0.
9. sum(rate(...)) by (le) before histogram_quantile.
10. Widen the discovery window for sparse metrics; get-available-metrics only looks back 1 hour by default.
11. Cloud-provider metrics are all one series — kubesense_cloud_resource_metric — with the provider's metric name in the metric_name label. Discover them with count by (metric_name) (...), never get-available-metrics.
12. Never rate() a cloud metric, and always pin resource_type next to metric_name.