pyroscope

Grafana Pyroscope Skill

Comprehensive guide for Grafana Pyroscope - the open-source continuous profiling platform for analyzing application performance at the code level.

What is Pyroscope?

Pyroscope is a horizontally-scalable, highly-available, multi-tenant continuous profiling system that:

• Collects profiling data continuously with minimal overhead (~2-5% CPU)
• Provides code-level visibility with source-line granularity
• Stores compressed profiles in object storage (S3, GCS, Azure Blob)
• Integrates with Grafana for correlating profiles with metrics, logs, and traces
• Supports multiple languages - Go, Java, Python, .NET, Ruby, Node.js, Rust

Architecture Overview

Core Components

Component	Purpose
Distributor	Validates and routes incoming profiles to ingesters
Ingester	Buffers profiles in memory, compresses and writes to storage
Querier	Retrieves and processes profile data for analysis
Query Frontend	Handles query requests, caching, and scheduling
Query Scheduler	Manages per-tenant query queues
Store Gateway	Provides access to long-term profile storage
Compactor	Merges blocks, manages retention, handles deletion

Data Flow

Write Path:

SDK/Alloy → Distributor → Ingester → Object Storage
                                   ↓
                             Blocks + Indexes

Read Path:

Query → Query Frontend → Query Scheduler → Querier
                                             ↓
                                    Ingesters + Store Gateway

Deployment Modes

1. Monolithic Mode (-target=all)

• All components in single process
• Best for: Development, small-scale deployments
• Query URL: http://pyroscope:4040/

2. Microservices Mode (Production)

• Each component runs independently
• Horizontally scalable
• Query URL: http://pyroscope-querier:4040/

# Microservices deployment
architecture:
  microservices:
    enabled: true

querier:
  replicas: 3
distributor:
  replicas: 2
ingester:
  replicas: 3
compactor:
  replicas: 3
storeGateway:
  replicas: 3

Quick Start - Kubernetes Helm

Add Repository

helm repo add grafana https://grafana.github.io/helm-charts
helm repo update

Install Single Binary

kubectl create namespace pyroscope
helm install pyroscope grafana/pyroscope -n pyroscope

Install Microservices Mode

curl -Lo values-micro-services.yaml \
  https://raw.githubusercontent.com/grafana/pyroscope/main/operations/pyroscope/helm/pyroscope/values-micro-services.yaml

helm install pyroscope grafana/pyroscope \
  -n pyroscope \
  --values values-micro-services.yaml

Profile Types

Type	Description	Languages
CPU	Wall/CPU time consumption	All
Memory	Allocation objects/space, heap	Go, Java, .NET
Goroutine	Concurrent goroutines	Go
Mutex	Lock contention (count/duration)	Go, Java, .NET
Block	Thread blocking/delays	Go
Exceptions	Exception tracking	Python

Client Configuration Methods

Method 1: SDK Instrumentation (Push Mode)

Go SDK:

import "github.com/grafana/pyroscope-go"

pyroscope.Start(pyroscope.Config{
    ApplicationName: "my-app",
    ServerAddress:   "http://pyroscope:4040",
    ProfileTypes: []pyroscope.ProfileType{
        pyroscope.ProfileCPU,
        pyroscope.ProfileAllocObjects,
        pyroscope.ProfileAllocSpace,
        pyroscope.ProfileInuseObjects,
        pyroscope.ProfileInuseSpace,
        pyroscope.ProfileGoroutines,
        pyroscope.ProfileMutexCount,
        pyroscope.ProfileMutexDuration,
        pyroscope.ProfileBlockCount,
        pyroscope.ProfileBlockDuration,
    },
    Tags: map[string]string{
        "env": "production",
    },
})

Java SDK:

PyroscopeAgent.start(
    new Config.Builder()
        .setApplicationName("my-app")
        .setServerAddress("http://pyroscope:4040")
        .setProfilingEvent(EventType.ITIMER)
        .setFormat(Format.JFR)
        .build()
);

Python SDK:

import pyroscope

pyroscope.configure(
    application_name="my-app",
    server_address="http://pyroscope:4040",
    tags={"env": "production"},
)

Method 2: Grafana Alloy (Pull Mode)

Auto-instrumentation via Annotations:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  template:
    metadata:
      annotations:
        profiles.grafana.com/cpu.scrape: "true"
        profiles.grafana.com/cpu.port: "8080"
        profiles.grafana.com/memory.scrape: "true"
        profiles.grafana.com/memory.port: "8080"
        profiles.grafana.com/goroutine.scrape: "true"
        profiles.grafana.com/goroutine.port: "8080"

Alloy Configuration:

pyroscope.scrape "default" {
  targets = discovery.kubernetes.pods.targets
  forward_to = [pyroscope.write.default.receiver]

  profiling_config {
    profile.process_cpu { enabled = true }
    profile.memory { enabled = true }
    profile.goroutine { enabled = true }
  }
}

pyroscope.write "default" {
  endpoint {
    url = "http://pyroscope:4040"
  }
}

Method 3: eBPF Profiling (Linux)

For compiled languages (C/C++, Go, Rust):

pyroscope.ebpf "default" {
  forward_to = [pyroscope.write.default.receiver]
  targets = discovery.kubernetes.pods.targets
}

Storage Configuration

Azure Blob Storage

pyroscope:
  config:
    storage:
      backend: azure
      azure:
        container_name: pyroscope-data
        account_name: mystorageaccount
        account_key: ${AZURE_ACCOUNT_KEY}

AWS S3

pyroscope:
  config:
    storage:
      backend: s3
      s3:
        bucket_name: pyroscope-data
        region: us-east-1
        endpoint: s3.us-east-1.amazonaws.com
        access_key_id: ${AWS_ACCESS_KEY_ID}
        secret_access_key: ${AWS_SECRET_ACCESS_KEY}

Google Cloud Storage

pyroscope:
  config:
    storage:
      backend: gcs
      gcs:
        bucket_name: pyroscope-data
        # Uses GOOGLE_APPLICATION_CREDENTIALS

Grafana Integration

Data Source Configuration

apiVersion: 1
datasources:
  - name: Pyroscope
    type: grafana-pyroscope-datasource
    access: proxy
    url: http://pyroscope-querier:4040
    isDefault: false
    editable: true

Trace-to-Profile Linking

Enable span profiles to correlate traces with profiles:

Go with OpenTelemetry:

import (
    "github.com/grafana/pyroscope-go"
    otelpyroscope "github.com/grafana/otel-profiling-go"
)

tp := trace.NewTracerProvider(
    trace.WithSpanProcessor(otelpyroscope.NewSpanProcessor()),
)

Requirements:

• Minimum span duration: 20ms
• Supported: Go, Java, .NET, Python, Ruby

Resource Requirements

Single Binary (Development)

resources:
  requests:
    cpu: 500m
    memory: 512Mi
  limits:
    cpu: 1
    memory: 2Gi

Microservices (Production)

Component	CPU Request	Memory Request	Memory Limit
Distributor	500m	256Mi	1Gi
Ingester	1	8Gi	16Gi
Querier	100m	256Mi	1Gi
Query Frontend	100m	256Mi	1Gi
Compactor	1	8Gi	16Gi
Store Gateway	1	8Gi	16Gi

Common Helm Values

# Production values
architecture:
  microservices:
    enabled: true

pyroscope:
  persistence:
    enabled: true
    size: 50Gi

  config:
    storage:
      backend: s3
      s3:
        bucket_name: pyroscope-prod
        region: us-east-1

# High availability
ingester:
  replicas: 3
  terminationGracePeriodSeconds: 600

querier:
  replicas: 3

distributor:
  replicas: 2

compactor:
  replicas: 3
  terminationGracePeriodSeconds: 1200

storeGateway:
  replicas: 3

# Pod disruption budget
podDisruptionBudget:
  enabled: true
  maxUnavailable: 1

# Topology spread
topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: kubernetes.io/hostname
    whenUnsatisfiable: DoNotSchedule

# Monitoring
serviceMonitor:
  enabled: true

# Alloy for profile collection
alloy:
  enabled: true

API Endpoints

Ingestion

# Push profiles (Connect API)
POST /push.v1.PusherService/Push

# Legacy HTTP (pprof, JFR formats)
POST /ingest

Query

# Merged profile
POST /querier.v1.QuerierService/SelectMergeProfile

# Flame graph data
POST /querier.v1.QuerierService/SelectMergeStacktraces

# Available labels
POST /querier.v1.QuerierService/LabelNames

# Profile types
POST /querier.v1.QuerierService/ProfileTypes

# Legacy render
GET /pyroscope/render?query={}&from=now-1h&until=now

System

# Readiness
GET /ready

# Configuration
GET /config

# Metrics
GET /metrics

Troubleshooting

Diagnostic Commands

# Check pod status
kubectl get pods -n pyroscope -l app.kubernetes.io/name=pyroscope

# View ingester logs
kubectl logs -n pyroscope -l app.kubernetes.io/component=ingester --tail=100

# Check ring status
kubectl exec -it pyroscope-0 -n pyroscope -- \
  curl http://localhost:4040/ingester/ring

# Verify readiness
kubectl exec -it pyroscope-0 -n pyroscope -- \
  curl http://localhost:4040/ready

# Check configuration
kubectl exec -it pyroscope-0 -n pyroscope -- \
  curl http://localhost:4040/config

Common Issues

1. Ingester OOM:

ingester:
  resources:
    limits:
      memory: 16Gi

2. Storage Authentication Failed:

# Azure - verify RBAC
az role assignment create \
  --role "Storage Blob Data Contributor" \
  --assignee-object-id <principal-id> \
  --scope <storage-scope>

3. High Cardinality Labels:

# Limit label cardinality
pyroscope:
  config:
    validation:
      max_label_names_per_series: 25

4. Query Timeout:

pyroscope:
  config:
    querier:
      query_timeout: 5m
      max_concurrent: 8

Reference Documentation

For detailed configuration by topic:

• Helm Deployment: Complete Helm values reference
• Architecture: Component details and scaling
• SDK Instrumentation: Language SDK guides
• Troubleshooting: Common issues and diagnostics

External Resources

• Official Pyroscope Documentation
• Pyroscope Helm Chart
• Pyroscope GitHub Repository
• Grafana Profiles Drilldown

---

Gotchas

• Profile sampling rate per-application: 100Hz on a busy service adds non-trivial overhead — profile in canary first.
• Storage compaction blocks new writes during merge — large compactions cause write stalls; chunk size and compaction window need tuning.
• Tag cardinality explosion: per-request tags balloon storage; cap tag values explicitly or storage doubles weekly.
• Trace integration: span → profile linking requires matching traceparent header propagation end-to-end; one un-instrumented hop loses linkage silently.
• eBPF profiler vs SDK profiler: different attribution, sometimes conflicting. Don't run both for the same service.
• gRPC vs HTTP push: same data, different rate limits at the ingester; HTTP push is throttled harder under load.