oodle-monitors

Oodle Monitors — CRUD and Alerting

This skill teaches the agent to build, validate, and update Oodle monitors so that alerts are actionable, scoped, and free from flapping.

Prerequisites

# Install + configure (see oodle-cli skill)
brew install oodle-ai/oodle/oodle
oodle configure
# or
export OODLE_API_KEY=<key>
export OODLE_INSTANCE=<instance>
export OODLE_DEPLOYMENT=<url>

Confirm authentication and that at least one monitor list call succeeds before creating new monitors:

oodle monitors list -o json | jq 'length'

Command Execution Order

Before running any oodle command:

1. Check whether the required resource ID or name is already in context.
2. If not, run the discovery command (e.g., oodle monitors list -o json).
3. If the result is ambiguous, ask the user to confirm before proceeding.
4. Run the target command with the resolved ID.
5. Do not run speculative commands (e.g., do not delete without first get-ing the resource).

Quick Reference

Task	Command
List all monitors	oodle monitors list -o json
Filter by status	oodle monitors list --status alert -o json
Filter by labels	oodle monitors list --labels env=prod,team=platform -o json
Get one monitor	oodle monitors get <id> -o json
Create from file	oodle monitors create -f monitor.json
Update from file	oodle monitors update <id> -f monitor.json
Delete (CI)	oodle monitors delete <id> --force

Common Operations

Listing monitors

# ✅ CORRECT — JSON output for scripting
oodle monitors list -o json

# ✅ CORRECT — narrow with --status to find only firing alerts
oodle monitors list --status alert -o json

# ✅ CORRECT — narrow with --labels for a specific team
oodle monitors list --labels env=prod,team=platform -o json

# ❌ WRONG — pulling everything then grepping
oodle monitors list | grep CPU

Reading a monitor before changing it

# ✅ CORRECT — fetch full JSON, edit, then update
oodle monitors get mon_123 -o yaml > monitor.yaml
$EDITOR monitor.yaml
oodle monitors update mon_123 -f monitor.yaml

# ❌ WRONG — building update payload from memory; overwrites unrelated fields
oodle monitors update mon_123 -f <(echo '{"options":{"thresholds":{"critical":90}}}')

Creating a monitor

A complete, valid monitor JSON:

{
  "name": "High CPU on web servers",
  "type": "metric alert",
  "query": "avg(last_5m):avg:system.cpu.user{env:prod,service:api} by {host} > 80",
  "message": "CPU above 80% on {{host.name}}. Runbook: https://runbooks.example.com/cpu\n@slack-ops",
  "labels": {"team": "platform", "env": "prod"},
  "options": {
    "thresholds": {
      "critical": 80,
      "critical_recovery": 70,
      "warning": 60,
      "warning_recovery": 50
    }
  }
}

# ✅ CORRECT
oodle monitors create -f monitor.json

# ❌ WRONG — no `type`, no `options.thresholds`, monitor will be rejected
oodle monitors create -f <(echo '{"name":"x","query":"y"}')

Updating a monitor

# ✅ CORRECT — get → edit → update
oodle monitors get mon_123 -o json > monitor.json
jq '.options.thresholds.critical = 85' monitor.json > monitor.new.json
oodle monitors update mon_123 -f monitor.new.json

# ❌ WRONG — sending only the changed field; missing fields become null
oodle monitors update mon_123 -f <(echo '{"options":{"thresholds":{"critical":85}}}')

Deleting a monitor

# ✅ CORRECT — verify first, then delete
oodle monitors get mon_123 -o json > /dev/null
oodle monitors delete mon_123 --force

# ❌ WRONG — speculative delete by name match
oodle monitors delete "$(oodle monitors list | grep CPU | head -1 | awk '{print $1}')" --force

Best Practices

Use a last_5m (or longer) evaluation window, not last_1m

Short windows cause alert flapping on normal traffic spikes. Use last_5m minimum for production alerts, last_15m for noisy metrics.

# ✅ CORRECT
"query": "avg(last_5m):avg:system.cpu.user{env:prod} by {host} > 80"

# ❌ WRONG — flaps on every brief spike
"query": "avg(last_1m):avg:system.cpu.user{env:prod} by {host} > 80"

Scope queries with explicit labels — never use {*}

{*} matches every series in the system and produces alerts for resources you don't own.

# ✅ CORRECT — scoped to a specific env + service
"query": "avg(last_5m):avg:system.cpu.user{env:prod,service:api} by {host} > 80"

# ❌ WRONG — alerts on every host in the org
"query": "avg(last_5m):avg:system.cpu.user{*} > 80"

Always set *_recovery thresholds below the trigger thresholds

Without recovery thresholds the monitor stays in alert state until the metric drops below the critical threshold exactly — small oscillations keep the alert active forever.

# ✅ CORRECT — clear recovery band (10pt below trigger)
"thresholds": {"critical": 80, "critical_recovery": 70, "warning": 60, "warning_recovery": 50}

# ❌ WRONG — no recovery values; monitor never cleanly recovers
"thresholds": {"critical": 80, "warning": 60}

Put a runbook URL and an @notifier handle in message

Alerts without an action are noise. Every monitor message must answer "what do I do?" and "who is paged?".

# ✅ CORRECT
"message": "CPU above 80% on {{host.name}} (env=prod, service=api).\nRunbook: https://runbooks.example.com/cpu\n@slack-ops @pagerduty-platform"

# ❌ WRONG — no actionable content, no routing
"message": "CPU is high"

Tag every monitor with at least team and env labels

Labels are how notification policies route alerts and how oodle monitors list --labels ... filters work.

# ✅ CORRECT
"labels": {"team": "platform", "env": "prod", "service": "api"}

# ❌ WRONG
"labels": {}

Failure Handling

Error	Cause	Fix
401 Unauthorized	Invalid or missing API key	Run oodle configure or set OODLE_API_KEY
404 Not Found	Monitor ID does not exist	Verify with oodle monitors list -o json
connection refused	Wrong OODLE_DEPLOYMENT URL	Check OODLE_DEPLOYMENT env var
invalid query	PromQL/Datadog-style query has a syntax error	Test the query in the UI metrics explorer; ensure avg(last_5m): prefix and by {label} suffix
Alert never fires	Query returns no data, or threshold is unreachable	Run oodle metrics list --match <metric> to confirm the metric exists; lower the threshold temporarily and re-check
Too many alerts (flapping)	Evaluation window too short, missing recovery thresholds	Increase window to last_5m or last_15m; set critical_recovery and warning_recovery
no data alerts	Agent not reporting, or label filter excludes all hosts	Verify the agent is alive (oodle metrics list --match up); widen the label filter to confirm any series match
429 Too Many Requests	Bulk monitor creation hit rate limit	Add --retries 3, throttle to <10 creates per second

References

• Oodle CLI repo
• Oodle docs