Prometheus: From Metrics to PromQL
Explore a living Prometheus monitoring system over the persistent ACME Shop microservice architecture. Trace metric exposition, target discovery, relabeling pipelines, TSDB Head and WAL internals, PromQL AST evaluation, Alertmanager routing, and live incident triage.
The Prometheus Mental Model: Pull-Based Telemetry Over ACME Shop
How does Prometheus discover, pull, and transform telemetry without active agent pushing?
Core Mental Model: Prometheus periodically pulls (scrapes) HTTP endpoints exposed by applications, indexing multidimensional numeric samples into an append-only TSDB queried via PromQL.
Operational Boundary: Prometheus is an operational monitoring system, not an event-logging bus or transactional ledger. It records numeric values over time with millisecond timestamps, not arbitrary text payloads.
Metric Anatomy: Labels, Series Identity, and Sample Values
What constitutes a unique time series in Prometheus TSDB?
METRIC NAME: http_requests_total
Specifies the general feature of a system that is measured (e.g. http_requests_total for cumulative HTTP requests). By Prometheus convention, counter names end in _total and units like _seconds or _bytes are included in the base name.
CONSECUTIVE SCRAPES TIMELINE (15s SCRAPE INTERVAL)
Metric Name + Exact Set of Key-Value Labels = Unique Time Series ID. Altering, adding, or deleting a single label creates a completely new time series with its own independent memory allocation in TSDB.
Core Mental Model: A time series is uniquely identified by the combination of its metric name and full key-value label set: {metric_name, label1=val1, label2=val2}. Adding even one label creates a completely new series.
Operational Boundary: Every distinct combination of label key-value pairs allocates a separate in-memory series header in the TSDB Head chunk. Labels are dimensions, not free-form logging annotations.
Metric Types Laboratory: Counter, Gauge, Histogram, Summary
When must an engineer choose a Counter versus a Gauge, Histogram, or Summary?
RAW COUNTER VALUE: http_requests_total
Raw counter drops to 0 when process restarts. Always apply rate() or increase() for graphing.
DERIVED RATE: rate(http_requests_total[5m])
rate() automatically detects counter resets and compensates by adding the pre-reset value.
AVAILABLE OVER-TIME OPERATORS (NEVER USE rate() ON A GAUGE):
avg_over_time(active_connections[1h])Mean connection levelmax_over_time(active_connections[1h])Peak concurrency spikemin_over_time(active_connections[1h])Lowest troughpredict_linear(node_filesystem_free_bytes[4h], 86400)Linear exhaustion extrapolationhistogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket[5m])))Calculates the 95th percentile across all instances by linearly interpolating between bucket boundaries le="0.1" and le="0.25".
HISTOGRAM (SERVER-SIDE AGGREGATABLE)
- Buckets recorded independently per instance.
- Quantiles mathematically aggregatable across 100 instances via
histogram_quantile(). - High-cardinality overhead (10–20 series per metric).
- Bucket boundaries must be configured ahead of time.
SUMMARY (CLIENT-SIDE SLIDING WINDOW)
- Quantiles (P50, P90, P99) calculated directly inside application RAM.
- CANNOT BE AGGREGATED: You cannot average two P95 percentiles together mathematically!
- Low time-series footprint (only 3–4 series per instance).
- CPU and memory overhead on client application runtime.
Core Mental Model: Counters only increment (or reset on process restart) and track totals. Gauges can rise and fall to represent current state. Histograms aggregate observations into configurable cumulative buckets. Summaries calculate client-side quantiles.
Operational Boundary: Never calculate rate() on a Gauge—rate() assumes counter reset semantics and will produce nonsensical negative spikes or reset compensation artifacts.
High-Resolution Latency: Classic Cumulative Buckets vs Native Histograms
Why do classic histograms suffer from bucket boundary estimation errors and cardinality bloat?
17 INDEPENDENT TIME SERIES CREATED
http_request_duration_seconds_bucket{le="0.005"}http_request_duration_seconds_bucket{le="0.01"}http_request_duration_seconds_bucket{le="0.025"}http_request_duration_seconds_bucket{le="0.05"}... 12 more le bucket series ...http_request_duration_seconds_sumhttp_request_duration_seconds_count1 SINGLE COMPACT TIME SERIES
Core Mental Model: Classic histograms store each bucket as an independent time series with an le label (typically 10–20 series per metric). Native histograms store an entire dynamic exponential distribution inside a single TSDB sample.
Operational Boundary: Native histograms are stable in Prometheus 3.8+, but your client library and remote write endpoints must explicitly support sparse exponential histogram schemas (e.g. scrape_native_histograms: true).
Cardinality Multiplier: The Physics of TSDB Head Memory Exhaustion
How does an innocent label like user_id cause Prometheus to OOM within minutes?
ACTIVE LABEL DIMENSIONS
user_id, email, request_id, jwt_token, uuid, raw_url_with_query_paramsCore Mental Model: Total series count is the Cartesian product of all distinct label values across every monitored target. Unbounded labels (UUIDs, user IDs, raw URLs) explode memory exponentially.
Operational Boundary: Each active series consumes ~1.5 KB of RAM in the TSDB Head chunk. Adding a label with 100,000 values multiplies existing series by 100,000, quickly consuming 15–30 GB of RAM.
The Scrape Journey: From Service Discovery to TSDB Ingestion
What exact stages does a scrape request undergo from discovery to memory storage?
STAGE 1: SERVICE DISCOVERY
Prometheus reaches out to the platform provider (Kubernetes API server, AWS EC2 API, Consul, or file_sd) to discover active workload IP addresses and ports, receiving raw metadata prefixed with __meta_*.
__address__ = "10.244.2.45:8080" __meta_kubernetes_namespace = "prod" __meta_kubernetes_pod_name = "checkout-7d9c89f" __meta_kubernetes_pod_label_app = "checkout"
Core Mental Model: The scrape loop moves through: Discovery -> Target Relabeling -> HTTP GET /metrics -> Protocol Parsing -> Metric Relabeling -> TSDB Appender -> WAL + Head chunk.
Operational Boundary: Scrapes occur strictly on the configured scrape_interval timer. If a scrape takes longer than scrape_timeout, the scrape is aborted, the sample is marked stale, and up evaluates to 0.
Service Discovery: Static, DNS, Kubernetes, EC2, Consul, and File SD
How does Prometheus know which IP addresses and ports to monitor in dynamic cloud environments?
PROMETHEUS CONFIGURATION (prometheus.yml)
scrape_configs:
- job_name: 'kubernetes-pods'
kubernetes_sd_configs:
- role: pod
namespaces:
names: ['prod']
relabel_configs:
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
action: keep
regex: trueDISCOVERED TARGET METADATA POOL
| Target Address | Namespace | Pod / Service | Action |
|---|---|---|---|
| 10.244.1.12:8080 | prod | checkout-1 | SCRAPED |
| 10.244.2.33:8080 | prod | checkout-2 | SCRAPED |
| 10.244.3.18:8080 | prod | checkout-3 | SCRAPED |
| 10.244.1.99:9090 | kube-system | coredns | DROPPED |
Core Mental Model: Prometheus queries platform APIs (Kubernetes API, AWS EC2, Consul, DNS SRV) to stream target metadata prefixed with __meta_*, which relabeling then transforms into operational labels.
Operational Boundary: Prometheus never listens for targets to register themselves; it continuously polls discovery providers to maintain an in-memory target pool, maintaining pull architecture sovereignty.
Relabeling Laboratory: The Rule Engine Powering Target Metadata Transformation
How do source_labels, regex, action, and target_label rewrite dimensions on the fly?
INPUT DISCOVERED METADATA
RELABEL PIPELINE RULES
source_labels: [__meta_kubernetes_pod_label_app]target_label: jobsource_labels: [__address__]target_label: instanceFINAL TARGET LABELS (INGESTED)
Core Mental Model: Relabeling is a sequential filter pipeline applied to target metadata (before scrape) or scraped samples (after scrape). Actions include replace, keep, drop, labelmap, labeldrop, and hashmod.
Operational Boundary: A failed keep or matched drop rule discards the entire target or sample silently. Always test relabel regexes with promtool or this interactive visual laboratory before applying to production.
Target Relabeling vs Metric Relabeling: Architectural Timing and Scope
What is the critical operational difference between relabel_configs and metric_relabel_configs?
relabel_configs (Target Relabeling)
Operates on target metadata (__meta_* and __address__) discovered from provider APIs.
If action: drop matches, the entire target is omitted from the scrape pool. ZERO network HTTP requests occur!
metric_relabel_configs (Metric Relabeling)
Operates on individual metric lines parsed from the target's HTTP /metrics response body.
If action: drop matches on __name__="debug_.*", the metric is discarded before TSDB append, but the HTTP scrape already consumed bandwidth.
Core Mental Model: Target relabeling (relabel_configs) executes BEFORE the HTTP scrape to determine WHICH targets exist and their target-level labels. Metric relabeling (metric_relabel_configs) executes AFTER HTTP parsing to filter or drop specific metric samples before TSDB ingestion.
Operational Boundary: Dropping a high-cardinality metric in metric_relabel_configs saves TSDB storage and RAM, but does NOT reduce network bandwidth or exporter CPU because the metric is still scraped across the wire.
Prometheus Target Inspector: Health Checks, Scrape Latency, and Synthetic "up" Metric
How does Prometheus record target availability and scrape duration?
PROMETHEUS TARGETS POOL (10 / 11 HEALTHY)
Core Mental Model: On every scrape, Prometheus appends two synthetic metrics: up{job="...", instance="..."} (1 for success, 0 for failure) and scrape_duration_seconds (time taken to complete HTTP scrape).
Operational Boundary: When a target goes down, Prometheus does not erase its historical data. It appends up=0 and a stale NaN marker to indicate the series has ceased publishing.
TSDB Internals: Head, WAL, Chunks, and Inverted Index Architecture
Where do samples live in memory before they are written to durable on-disk blocks?
http_requests_total{service="checkout"} 18421.0 @ 1726754535000DURABLE WRITE-AHEAD LOG (WAL)
data/wal/00000125 (128 MB segment)Append-only record of new series headers and samples. Flushed to disk synchronously so uncompacted data survives process crashes.
IN-MEMORY HEAD CHUNK
data/chunks_head/Active chunk buffers in process RAM. Once filled (~120 samples), chunks are cut and memory-mapped (mmap) to disk, freeing RAM.
IMMUTABLE 2-HOUR BLOCK (e.g. 01J8F9X...)
Core Mental Model: New samples are appended concurrently to the Write-Ahead Log (WAL) on disk and the in-memory Head chunk. Once Head chunks fill (~120 samples), they are cut to mmap-backed chunks, and every 2 hours flushed to immutable blocks.
Operational Boundary: Head chunks reside in process memory. If Prometheus crashes, uncompacted Head chunks are lost from RAM and must be reconstructed by replaying WAL log segments on restart.
TSDB Time Horizon: 2-Hour Blocks, Compaction Waves, and Retention Windows
How does Prometheus prevent disk fragmentation across weeks of continuous metric collection?
Core Mental Model: Prometheus writes immutable 2-hour blocks consisting of chunks, index, meta.json, and tombstones. A background compaction loop merges adjacent 2h blocks into larger 6h, 18h, and 54h blocks, removing soft-deleted tombstones.
Operational Boundary: Compaction creates temporary duplicate blocks during merges, requiring up to 30% additional scratch disk headroom. Never run Prometheus on a disk with >85% baseline utilization.
Crash Recovery Simulation: How the Write-Ahead Log Prevents Telemetry Loss
What happens when Prometheus is killed with SIGKILL or suffers host power loss?
PROCESS MEMORY (HEAD CHUNK)
Unpersisted samples currently buffered in active Head memory.
DISK STORAGE (WAL + BLOCKS)
data/wal/00000124 (128 MB)data/wal/00000125 (42 MB)[STATUS] Prometheus running normally. WAL segments receiving new sample append operations.Core Mental Model: The WAL records every append operation in durable 128 MB disk segments. On startup, Prometheus iterates through existing immutable blocks, replays WAL segments since the last checkpoint, and reconstructs the active Head state.
Operational Boundary: Corrupted WAL segments from sudden disk power failures can stall Prometheus boot. Modern Prometheus can auto-repair minor corruptions, but catastrophic storage failure requires manual promtool recovery.
Storage Sizing Calculator: Gorilla Chunk Compression & Disk Planning
How do you accurately estimate disk requirements for 2 million active series?
CAPACITY SIZING SLIDERS
ESTIMATED LOCAL DISK REQUIREMENTS
Educational Estimate: Actual requirements vary based on timestamp jitter, churn rate, and label cardinality.
Core Mental Model: Prometheus utilizes Gorilla double-delta timestamp and XOR value compression, averaging 1.3 to 2.0 bytes per stored sample: Disk = Retention(sec) x ScrapeRate(samples/sec) x 1.7 bytes + CompactionBuffer(30%).
Operational Boundary: High churn (frequent pod restarts, short-lived jobs) degrades compression efficiency toward 3+ bytes/sample due to repeated series header postings in the inverted index.
The PromQL Dojo: Interactive Query Engine & Visual Transformation Pipeline
How does PromQL evaluate an expression from raw time series down to a final graph or table?
| Metric / Labels | Sample Value |
|---|---|
{service="catalog"} | 210.45 req/s |
{service="checkout"} | 139.80 req/s |
{service="payment"} | 84.12 req/s |
Core Mental Model: PromQL evaluates queries as a pipelined dataflow: Label Filtering -> Range Vector Windowing -> Mathematical Transformation (rate/increase) -> Grouped Aggregation (sum/avg) -> Result Vector.
Operational Boundary: PromQL does not return arbitrary table schemas. Every expression evaluates to one of four fundamental types: Instant vector, Range vector, Scalar, or String.
Selectors and Matchers: Querying Series via Equality and RE2 Regex Filters
How do equality and regular expression label matchers narrow down time series?
http_requests_total{status=~"5.."}Google RE2 regex match. Fully anchored (^ and $) implicitly. Matches status codes starting with 5 (500, 502, 503, 504).
{service="checkout", status="500"}Status starts with 5{service="payment", status="503"}Status starts with 5{service="checkout", status="200"}Status does not match 5..{service="catalog", status="404"}Status does not match 5..Core Mental Model: PromQL supports four matchers: = (exact match), != (not equal), =~ (regex match), and !~ (regex not match). Regexes use Google RE2 syntax and are fully anchored by default (^ and $ are implicit).
Operational Boundary: Unconstrained label matchers like {status=~".*"} force Prometheus to scan millions of postings lists in the index. Always supply at least one high-selectivity metric or label matcher.
Range Vectors vs Instant Vectors: Lookback Windows, offset, and Subqueries
Why cannot you graph a range vector directly in Grafana?
http_requests_total
A set of time series containing a single sample per series at the evaluation timestamp t.
http_requests_total[5m]
A set of time series containing a buffer of samples over the 5-minute lookback duration.
SUBQUERIES: rate(http_requests_total[5m])[30m:1m]
Evaluates the instant vector expression rate(...[5m]) at 1-minute steps over a 30-minute historical window, returning a derived range vector for secondary transformations like max_over_time().
Core Mental Model: An Instant Vector contains a single sample per series at evaluation time. A Range Vector (e.g. [5m]) contains a matrix of samples over a lookback duration, designed as input to functions like rate() or avg_over_time().
Operational Boundary: Graphing engines like Grafana require instant vectors at every step interval. Attempting to plot http_requests_total[5m] directly throws a type error because a graph point cannot plot an array of points.
Counter Mechanics: rate(), irate(), and Automatic Counter Reset Handling
How does rate() accurately measure requests per second even when an application restarts?
rate(v[5m])RECOMMENDED FOR ALERTSCalculates the average per-second rate of increase over the entire range window. Smooths out momentary spikes and extrapolates boundaries.
Smooth, reliable trend line for SLO and alert evaluations.irate(v[5m])DO NOT USE IN ALERTSCalculates the instant rate of increase using ONLY the last two data points in the window. Highly sensitive to micro-bursts.
Volatile; can trigger alert flapping on transient network delays.increase(v[5m])TOTAL VALUE DELTACalculates total increase over the window. Equivalent to rate(v[5m]) * 300s. Compensates for counter resets.
Core Mental Model: rate() calculates the per-second average rate of increase across a range vector, extrapolating boundaries and automatically detecting when a counter resets (drops to 0) to add the previous value.
Operational Boundary: Never use irate() in alerting rules—irate() only inspects the last two samples and is extremely volatile, causing erratic alert firing on momentary network micro-bursts. Use rate() for alerts.
Aggregation Mechanics: sum, avg, min, max with "by" and "without" Clauses
How do you aggregate across instances without losing essential dimension tags?
INPUT SERIES (6 SERIES)
{service="checkout", instance="c1", status="200"} 45/s{service="checkout", instance="c2", status="200"} 48/s{service="checkout", instance="c3", status="500"} 4/s{service="catalog", instance="k1", status="200"} 110/s{service="catalog", instance="k2", status="200"} 100/s{service="payment", instance="p1", status="200"} 85/sOUTPUT SERIES
{service="catalog"} 210/s{service="checkout"} 97/s{service="payment"} 85/sCore Mental Model: Aggregation operators (sum, avg, count, min, max, topk) combine multiple series into fewer series. "by (label...)" preserves only the specified dimensions, while "without (label...)" preserves everything except the excluded labels.
Operational Boundary: Omitting both by and without collapses ALL series into a single dimensionless scalar series: sum(http_requests_total) strips service, instance, and status tags completely.
Vector Matching: One-to-One, Many-to-One, on(), ignoring(), and group_left
How does Prometheus perform mathematical joins between two distinct metric series?
LEFT VECTOR (ERRORS)
RIGHT VECTOR (REQUESTS)
Core Mental Model: Binary operations (e.g. errors / requests) require matching label sets. Use on(label...) to match on specific keys, ignoring(label...) to exclude mismatching keys, and group_left/group_right for many-to-one joins.
Operational Boundary: In group_left, the "many" side is on the LEFT and the "one" side is on the RIGHT. The left vector provides the resulting series identity while extra labels from the right vector can be copied.
Why Many-to-Many Joins Fail: Visualizing Ambiguous Cardinality in PromQL
Why does PromQL immediately abort with "many-to-many matching not allowed"?
LEFT VECTOR (DUPLICATE SERVICE)
RIGHT VECTOR (DUPLICATE SERVICE)
Prometheus refuses Cartesian joins because they create ambiguous identity. Resolve by aggregating one side with sum by (service) or specifying group_left with unique discriminator keys.
Core Mental Model: Prometheus requires deterministic arithmetic. If both sides of a binary operation contain duplicate matching label sets, a Cartesian cross-product would occur, which PromQL explicitly forbids.
Operational Boundary: To resolve many-to-many errors, you must either aggregate one side down to unique keys using sum by (...) or identify missing discriminator labels using on(...) or ignoring(...).
Calculating Latency SLOs: Inside histogram_quantile() and Linear Interpolation
How does Prometheus mathematically derive P95 latency from cumulative bucket counts?
STEP-BY-STEP CALCULATION:
- Sum rates across instances: Aggregates cumulative counts per bucket boundary
le. - Find target rank: Rank = 0.95 * total_count = 0.95 * 1045 = 992.75.
- Locate containing bucket:
le="0.25"(820) tole="0.5"(960) → contains 95th percentile! - Linear interpolation: Assumes uniform distribution within the bucket to calculate precise 412ms estimate.
CUMULATIVE BUCKET DISTRIBUTION
Core Mental Model: histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket[5m]))) identifies which cumulative bucket contains the 95th percentile rank and linearly interpolates within that bucket boundary.
Operational Boundary: Histogram quantiles are approximations. If your buckets are too wide (e.g. 0.1s to 10s), linear interpolation assumes an even distribution within the bucket, skewing true percentile accuracy.
Operational PromQL Cheatsheet: Real Queries for Production Engineering
What are the battle-tested PromQL formulas for Traffic, Errors, Latency, Saturation, and K8s?
Find targets currently down
up == 0Lists all targets failing HTTP scrapes with connection refused, DNS error, or timeout.
Target availability percentage
avg_over_time(up[1h]) * 100Calculates the percentage of successful scrapes over the past hour.
Detect missing critical metric
absent(http_requests_total{service="checkout"})Returns a 1-value series if the specified metric has ceased reporting entirely.
Total cluster requests/sec
sum(rate(http_requests_total[5m]))Overall system request volume across all applications.
Throughput grouped by service
sum by (service) (rate(http_requests_total[5m]))Collapses instance series to give per-service request rates.
Top 5 endpoints by traffic
topk(5, sum by (endpoint) (rate(http_requests_total[5m])))Identifies the heaviest traffic endpoints in the cluster.
Total 5xx error rate/sec
sum(rate(http_requests_total{status=~"5.."}[5m]))Total rate of internal server errors across all services.
Cluster 5xx error percentage
100 * sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m]))Calculates the ratio of errors to total requests as a percentage.
Services breaching 1% error budget
(sum by (service) (rate(http_requests_total{status=~"5.."}[5m])) / sum by (service) (rate(http_requests_total[5m]))) * 100 > 1Filters services currently violating an SLO of 99% success rate.
P95 latency per service
histogram_quantile(0.95, sum by (le, service) (rate(http_request_duration_seconds_bucket[5m])))Calculates 95th percentile latency segmented by service.
P99 latency cluster-wide
histogram_quantile(0.99, sum by (le) (rate(http_request_duration_seconds_bucket[5m])))Measures high-tail latency across all requests.
Average request duration
rate(http_request_duration_seconds_sum[5m]) / rate(http_request_duration_seconds_count[5m])Calculates true mathematical average duration without histogram bucket approximations.
Container CPU cores used
sum by (pod) (rate(container_cpu_usage_seconds_total[5m]))Real-time CPU core utilization per Kubernetes pod.
Node memory usage percent
(1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100Accurate Linux memory saturation accounting for kernel caches.
Disk space remaining < 15%
(node_filesystem_avail_bytes / node_filesystem_size_bytes) * 100 < 15Fires when any mounted filesystem has less than 15% free space.
Compare throughput to 1 week ago
sum(rate(http_requests_total[5m])) / sum(rate(http_requests_total[5m] offset 1w))Ratio of current traffic compared to the identical day and hour last week.
TSDB Head active series count
prometheus_tsdb_head_seriesInternal gauge tracking total time series currently resident in memory.
Remote write pending backlog
prometheus_remote_storage_samples_pendingQueue depth of samples awaiting dispatch to remote storage.
Core Mental Model: High-performing SREs organize PromQL by operational intent rather than alphabetical functions: USE method (Utilization, Saturation, Errors) and Google Golden Signals (Latency, Traffic, Errors, Saturation).
Operational Boundary: Copying arbitrary PromQL from internet posts without checking scrape intervals can cause empty graphs. A range window [1m] with a 1m scrape interval fails because rate() requires at least 2 samples.
PromQL Dojo Practice Engine: 14 Levels of Scenario-Based SRE Challenges
Can you solve realistic production monitoring challenges under live system conditions?
Select All Checkout Requests
You are investigating traffic reaching the Checkout service.
Write a selector to query http_requests_total for the "checkout" service.
Need a hint?
- Specify the metric name followed by label matchers in curly braces {}.
- Use service="checkout".
Core Mental Model: Mastery comes from diagnosing real system scenarios. The practice engine validates semantic query intent against parsed AST rules rather than rigid character matching.
Operational Boundary: Challenges test operational reasoning: distinguishing error ratios from absolute error counts, detecting pod CPU throttling, and calculating multi-dimensional service availability.
Recording Rules: Precomputing Expensive PromQL Expressions Into New Time Series
How do you prevent 50 Grafana dashboards from overloading Prometheus with repeated queries?
sum by (service) (rate(http_requests_total[5m]))
service:http_requests:rate5m
RECORDING RULE SPECIFICATION (rules.yml)
groups:
- name: acme_service_rates
interval: 15s
rules:
- record: service:http_requests:rate5m
expr: sum by (service) (rate(http_requests_total[5m]))Core Mental Model: Recording rules evaluate complex PromQL expressions on a regular schedule (e.g. every 1m) and save the result as a brand new, highly efficient time series in TSDB.
Operational Boundary: Follow the standard Prometheus naming convention: level:metric:operations (e.g. job:http_requests_total:rate5m). Recording rules run in evaluation groups and consume disk space like scraped series.
Alerting Rules: The Inactive -> Pending -> Firing Lifecycle and the "for" Clause
How does Prometheus prevent transient network blips from paging engineers in the middle of the night?
- alert: HighErrorRate
expr: sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) > 0.05
for: 5m
labels:
severity: critical
team: checkoutError rate <= 5%. Normal operation.
Threshold breached. Timer counting: 00:00 / 05:00
Dispatched to Alertmanager on port 9093.
Core Mental Model: Alert rules evaluate PromQL boolean expressions. When true, the alert enters PENDING state. Only if the expression remains true continuously for the configured for: duration does it transition to FIRING and notify Alertmanager.
Operational Boundary: If the expression flickers false for even a single evaluation interval, the pending timer resets to 0. Use rate() lookback windows appropriately to smooth noisy thresholds.
Alertmanager Routing Trees: Hierarchy, Grouping, and Notification Dispatch
How does Alertmanager direct critical payment alerts to PagerDuty while routing warnings to Slack?
Core Mental Model: Alertmanager processes firing alerts through a hierarchical routing tree. An alert enters at the root route and evaluates child matchers (severity, team, service) until a leaf receiver is selected, with optional continue: true propagation.
Operational Boundary: If an alert does not match any child route, it falls back to the root route default receiver. Always define a sensible catch-all receiver on the root route to avoid silent alert drops.
Alertmanager Superpowers: Deduplication, Grouping, Active Silences, and Inhibition Rules
How do you prevent 100 alerts from firing when a single underlying Kubernetes node dies?
Batching Alerts into Digests
Combines alerts sharing identical group_by labels into a single notification.
group_wait: 30s (Initial delay for batching)group_interval: 5m (Time before sending new alerts)repeat_interval: 4h (Re-notification frequency)Suppressing Cascading Failures
Mutes a set of alerts if another matching alert is already firing.
NodeDown{node="w1"}→ INHIBITS →PodCrashLoop{node="w1"}Planned Maintenance Muting
Time-based mutes configured via UI or API without modifying config files.
alertname=~"HighLatency", env="staging"Active: 2026-09-20 14:00 – 16:00 UTCCore Mental Model: Inhibition rules automatically mute target alerts when a matching source alert is active (e.g. NodeDown inhibits ContainerCrashLoopBackOff). Grouping aggregates multiple related alerts into a single batch notification.
Operational Boundary: Inhibition rules require matching label equality (e.g. equal: [node, cluster]). If label names differ between exporter and kube-state-metrics, inhibition will fail to match.
Prometheus High Availability: Independent Parallel Scraping and Alert Deduplication
Why does Prometheus HA avoid active-passive clustering or shared database replication?
PROMETHEUS SERVER A
Independent TSDB + ScrapesEvaluates Alert RulesPROMETHEUS SERVER B
Independent TSDB + ScrapesEvaluates Alert RulesCore Mental Model: Prometheus HA consists of two identical, independent Prometheus servers scraping the exact same targets. Alertmanager handles deduplication of firing alerts received from both instances.
Operational Boundary: Prometheus instances do NOT replicate local TSDB blocks or synchronize WALs. Their graphs may differ slightly due to millisecond jitter in scrape timing.
Federation: Hierarchical Metric Aggregation Across Edge and Global Servers
How does a global Prometheus aggregate summaries from 20 regional data centers without overwhelming network links?
Central Monitoring Cluster
Scrapes summary metrics via /federate?match[]={job="cluster-summary"}
EDGE: US-EAST
5,000 Pods Scrapedjob:requests:rate5mEDGE: EU-CENTRAL
4,200 Pods Scrapedjob:requests:rate5mEDGE: AP-SOUTH
3,800 Pods Scrapedjob:requests:rate5mCore Mental Model: Federation allows a Prometheus server to scrape selected time series from another Prometheus server via the /federate HTTP endpoint using match[] query filters.
Operational Boundary: Never federate raw metrics or unbounded series like http_requests_total directly. Only federate pre-aggregated recording rule summaries (e.g. job:http_requests:rate5m).
Remote Write Architecture: Long-Term Storage, Sharding Queues, and Backlog Management
How does Prometheus ship millions of samples to Thanos, Cortex, M3DB, or VictoriaMetrics?
Core Mental Model: Remote Write reads samples from the TSDB Head via a memory queue, batches them into Snappy-compressed Protocol Buffer requests, and dispatches them across dynamic concurrent worker shards.
Operational Boundary: If the remote endpoint is slow or unreachable, the in-memory write queue backs up (prometheus_remote_storage_samples_pending). If memory capacity is exceeded, samples are dropped.
Prometheus Agent Mode: Lightweight Forwarding Without Local PromQL or Rule Engines
Why run a dedicated forwarding agent instead of a full Prometheus server at the edge?
Full Telemetry Engine
- ✓ Target Discovery & Scrape Engine
- ✓ Local Append-Only TSDB Storage
- ✓ 2-Hour Blocks & Compactor
- ✓ PromQL Engine (Instant & Range Queries)
- ✓ Recording & Alerting Rule Loops
Optimized Edge Forwarder
- ✓ Target Discovery & Scrape Engine
- ✓ Lightweight Local WAL (Crash Buffer)
- ✓ Remote Write 1.0 / 2.0 Streaming
- ✗ NO Local TSDB Blocks or Compaction
- ✗ NO PromQL Engine or HTTP Query API
- ✗ NO Alerting or Recording Rules
Core Mental Model: Prometheus Agent Mode (--enable-feature=agent) strips out local PromQL querying, alert rule evaluation, and historical block compaction, acting as a streamlined scrape-and-forward proxy with WAL protection.
Operational Boundary: Prometheus Agent cannot serve PromQL queries or trigger local alerts. If your remote storage is down longer than the agent WAL retention, edge data will be lost.
Kubernetes Telemetry: The Prometheus Operator Pattern and Declarative CRDs
How do ServiceMonitor, PodMonitor, and PrometheusRule CRDs automate scrape configuration?
DECLARATIVE CRDs (GitOps)
ServiceMonitorLabels: app=checkoutPodMonitorScrapes Pod ports directlyPrometheusRuleDefines alert expressionsReconciliation Controller
Compiles CRDs into prometheus.yml Secret. Issues HTTP POST /-/reload without downtime.
PROMETHEUS STATEFULSET
Live Scrapes RunningDynamic targets updatedCore Mental Model: The Prometheus Operator watches Kubernetes Custom Resource Definitions (CRDs) and dynamically reconciles the generated prometheus.yml config and rule files without server restarts.
Operational Boundary: A ServiceMonitor selects Kubernetes Services using label selectors, NOT Pods directly. If your Service selector does not match the target Service labels, Prometheus discovers 0 targets.
Modern Prometheus: OpenTelemetry (OTLP) Ingestion and Exemplar-Driven Tracing
How do modern SREs jump directly from a 99th percentile metric latency spike into a distributed trace?
Full UTF-8 Metric Support
Prometheus 3.0 supports arbitrary UTF-8 characters in metric names and label values using { "http.request.count", app.env="prod" } syntax.
Native Histograms
Exponential sparse bucketing in 1 TSDB sample. Reduces series cardinality by up to 80% while computing high-precision quantiles.
Remote Write 2.0
Metadata streaming, OpenTelemetry resource attribute passthrough, and improved Snappy wire efficiency.
Native OTLP Ingestion
Accepts OpenTelemetry metrics directly over HTTP/gRPC via /api/v1/otlp without requiring an external collector proxy.
Core Mental Model: Exemplars attach external trace identifiers (trace_id) to specific metric observations. Modern Prometheus can also receive OpenTelemetry metrics directly via the built-in OTLP receiver.
Operational Boundary: Exemplars are stored in a dedicated circular in-memory buffer in the TSDB Head and are not retained long-term in immutable TSDB blocks.
Troubleshooting Target Down: Diagnosing Connection Refused and Scrape Timeouts
What systematic diagnostic steps isolate why checkout-3 is reporting up == 0?
Target exists in K8s Endpoints: 10.244.3.18:8080
nc -vz 10.244.3.18 8080 → Connection Refused
kubectl describe pod checkout-3 → OOMKilled (Exit code 137)
Bump pod memory limit 512Mi → 1Gi. Kubelet restarts pod. Target returns to up == 1.
Core Mental Model: Follow the target triage ladder: 1. Service discovery presence -> 2. Network connectivity (nc/telnet) -> 3. HTTP path & port (/metrics) -> 4. Authentication/TLS -> 5. Exporter crash/hang.
Operational Boundary: A target reporting down does not always mean the application is dead; a slow /metrics handler exceeding scrape_timeout marks the target down even if HTTP traffic succeeds.
Troubleshooting Cardinality: Hunting Down Unbounded Labels and Memory Leaks
How do you pinpoint which label caused active series to spike from 200k to 6.9 million?
TOP 5 METRICS BY ACTIVE SERIES COUNT
| Metric Name | Series Count | Root Cause Label |
|---|---|---|
| http_requests_total | 2,840,000 | user_id (2.8M values) |
| process_cpu_seconds_total | 11 | none |
| http_request_duration_seconds | 48 | le |
EMERGENCY MITIGATION: metric_relabel_configs
metric_relabel_configs:
- action: labeldrop
regex: "user_id"✓ Drops user_id dimension at ingestion. Series drop from 2.8M to 480 series. TSDB Head RAM drops to 720 KB.
Core Mental Model: Query TSDB Head status via prometheus_tsdb_head_series or API /api/v1/status/tsdb to rank series by label cardinality, then apply metric_relabel_configs or deploy an application fix.
Operational Boundary: Increasing container memory limits is a temporary band-aid, not a fix; unbounded labels will eventually exhaust any memory allocation.
Troubleshooting Slow Queries: Execution Limits, Postings Scanning, and Rule Remedies
Why does sum(rate(http_requests_total[30d])) cause PromQL to time out or max out CPU?
sum(rate(http_requests_total[30d]))EXECUTION PROFILE
- Scanned 360 on-disk 2h TSDB blocks.
- Loaded 4,800,000 chunk samples into RAM.
- PromQL evaluation exceeded
query.timeout: 2m.
OPTIMIZED REMEDY
- Precompute with recording rule
service:http_requests:rate5m. - Query
sum_over_time(service:http_requests:rate5m[30d]). - Execution time: 3.2 ms (>35,000x faster).
Core Mental Model: Unconstrained 30-day regex queries force the query engine to load and scan tens of millions of raw chunk samples across dozens of compacted blocks from disk.
Operational Boundary: Set query.timeout and query.max-samples flags in production to prevent rogue ad-hoc dashboard queries from causing Prometheus server denial-of-service.
Troubleshooting Remote Write: Queue Starvation, Shard Exhaustion, and 429 Rate Limits
What causes prometheus_remote_storage_samples_pending to climb continuously?
Core Mental Model: Remote Write backlog occurs when ingestion rate exceeds network delivery throughput, commonly triggered by remote endpoint 429 rate-limiting, network throttling, or inadequate max_shards.
Operational Boundary: If the queue reaches capacity and samples are dropped, data cannot be recovered unless the remote endpoint catches up before the Head memory buffer drops them.
Troubleshooting Alert Storms: Suppressing Cascading Cascades via Alertmanager Inhibition
How do you stop on-call engineers from receiving 60 separate notifications during a network partition?
Availability Zone Switch Failure (us-east-1a)
WITHOUT INHIBITION RULES
On-call engineer bombarded with 142 pager buzzes!
WITH ALERTMANAGER INHIBITION
✓ Exactly 1 actionable page sent to On-Call!
Core Mental Model: Configure Alertmanager inhibit_rules to match common labels between a root failure (e.g. NodeDown or KubeNodeNotReady) and secondary symptoms (PodCrashLooping, HighLatency).
Operational Boundary: Always test inhibition rules in staging. Overly broad source matchers can inadvertently silence unrelated legitimate production alerts.
PromQL Boss Fight: Live Triage of a Multi-Service Checkout SLO Outage
Can you correlate throughput, error ratios, latency, and CPU saturation to locate root cause?
14:02 UTC — CHECKOUT SERVICE P95 LATENCY > 800ms
STEP 1: DETERMINE TOTAL CHECKOUT TRAFFIC
Measure total request throughput for Checkout to confirm if traffic surge is the cause:
Core Mental Model: Effective incident response uses the RED method (Rate, Errors, Duration) to isolate the failing component, drill down to the specific instance, and verify system recovery.
Operational Boundary: Do not jump to conclusions based on raw error counts alone; always calculate the error ratio relative to total throughput to avoid false positives during traffic swings.
Meta-Monitoring: Inside Prometheus Internal Metrics and Self-Observability
How do SREs monitor the health, memory, and scrape performance of Prometheus itself?
Core Mental Model: Prometheus dogfoods its own data model by scraping itself at /metrics. Key internal metrics include prometheus_tsdb_head_series, prometheus_rule_evaluation_duration_seconds, and scrape_duration_seconds.
Operational Boundary: If Prometheus crashes or deadlocks, it cannot evaluate alerts about its own death. Production infrastructure always uses an external watcher or secondary Prometheus server to meta-monitor.