Platform teams often have strong observability and a mature CI/CD setup, yet releases still depend on tribal knowledge or one-off dashboards. Service Level Objectives are a way to codify what “healthy enough to ship” means, and to apply that judgment automatically. In Kubernetes, platform SLOs (Service level objectives) translate raw cluster and edge metrics into simple, binary guardrails that gate promotions. The outcome is faster, safer changes with less debate: releases proceed when error budgets are healthy and pause when burn shows risk.
A platform SLO is a promise to tenants of the platform, not to end users of any single workload. It focuses on shared capabilities such as ingress reliability, deployment readiness, scheduling latency, and control plane responsiveness. A small set is sufficient. For example, “edge 99th percentile request latency below 500 ms” and “workload rollout success rate above 99%” cover a large fraction of incident classes that impact many teams. These objectives should be independent of any single application’s business logic and measurable using common signals exposed by the platform.
Signals should come from first-class, vendor-neutral sources. Kubernetes and its ecosystem expose high-quality telemetry via Prometheus metrics and OpenTelemetry pipelines. Ingress controllers, gateways, and service meshes emit request counts, latencies, and status codes. kube-state-metrics and the API server expose scheduling and rollout states. OpenTelemetry adds consistent application metrics where needed. The practical pattern is to standardize a handful of metric names and labels, and to provide adapters where components differ. Uniform labels such as environment, cluster, namespace, and version make SLO queries stable across teams.
Error budgets turn SLOs into operations. Pick a rolling window, usually 28–30 days, and define the allowed fraction of “bad” events within the objective. If the objective is 99.5% availability, the budget is 0.5% unavailability across the window. Burn-rate alerts catch dangerous trends much earlier than cumulative budget alone. Two windows in parallel, fast for sensitivity and slow for confidence, provide robust gating. If the short window burns many times faster than allowed and the long window also exceeds a lower multiple, freeze releases until conditions improve.
The following recording and alerting rules illustrate a neutral approach for an edge SLO using HTTP success rate. The same pattern applies to latency or rollout SLIs by swapping the numerator and denominator. The rules compute rolling success, derive budget burn for two windows, and raise a multi-window alert suitable for gating.
# PrometheusRule defining edge availability SLO with burn-rate alerts for release gating
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: platform-slo-edge
namespace: observability
spec:
groups:
- name: platform.edge.slo
interval: 30s
rules:
- record: slo:edge:request_success:ratio_rate5m
expr: |
1 - (
sum(rate(http_requests_total{job="edge",status=~"5.."}[5m]))
/
sum(rate(http_requests_total{job="edge"}[5m]))
)
- record: slo:edge:request_success:ratio_rate1h
expr: |
1 - (
sum(rate(http_requests_total{job="edge",status=~"5.."}[1h]))
/
sum(rate(http_requests_total{job="edge"}[1h]))
)
- record: slo:edge:availability:error_budget
labels:
slo: "edge-availability"
objective: "99.5"
expr: 1 - 0.995
- record: slo:edge:burn_rate5m
expr: (1 - slo:edge:request_success:ratio_rate5m) / slo:edge:availability:error_budget
- record: slo:edge:burn_rate1h
expr: (1 - slo:edge:request_success:ratio_rate1h) / slo:edge:availability:error_budget
- alert: PlatformEdgeErrorBudgetBurn
annotations:
summary: "Edge SLO burning error budget too fast"
runbook_url: "https://runbooks.internal/slo/edge"
expr: |
(slo:edge:burn_rate5m > 10 and slo:edge:burn_rate1h > 2)
for: 10m
labels:
severity: page
slo: "edge-availability"
guardrail: "release-freeze"
Release guardrails read these computed signals from the metrics API and decide whether to proceed. A gate should be explicit about the SLO it checks, the environment and cluster, and the thresholds. Favor hard-fail behavior with a narrowly scoped override mechanism to avoid noisy or subjective exceptions. When a gate fails, the pipeline should surface the burn rates, the current SLI, and a pointer to the runbook so engineers can act without context switching.
The minimal, portable way to integrate a gate is to query the Prometheus HTTP API from the pipeline and parse the result. The example below queries the one-hour burn and the five-minute burn for the edge SLO, compares against thresholds aligned with the alert above, and blocks on breach. It expects a Prometheus endpoint reachable from the runner and a bearer token or mTLS configured through the pipeline’s secret store.
# Bash script to query Prometheus burn rates and implement SLO-based release guardrails
#!/usr/bin/env bash
set -euo pipefail
PROM_URL="https://prometheus.platform.local"
AUTH="Authorization: Bearer ${PROM_TOKEN:?missing}"
Q5M='slo:edge:burn_rate5m'
Q1H='slo:edge:burn_rate1h'
THRESH_FAST=10
THRESH_SLOW=2
read_value () {
local query="$1"
curl -sS -H "$AUTH" --get --data-urlencode "query=${query}" "${PROM_URL}/api/v1/query" \
| jq -r '.data.result[0].value[1] // "NaN"'
}
fast=$(read_value "${Q5M}")
slow=$(read_value "${Q1H}")
echo "fast_window_burn=${fast}"
echo "slow_window_burn=${slow}"
awk -v f="$fast" -v s="$slow" -v tf="$THRESH_FAST" -v ts="$THRESH_SLOW" '
BEGIN {
if (f != f || s != s) { print "missing data"; exit 2 }
if (f > tf && s > ts) { print "error budget burning too fast"; exit 1 }
print "within guardrails"; exit 0
}'
Progressive delivery benefits from the same signals with an added filter on version. A canary or surge rollout should carry a stable label such as version and environment so that the SLI can be evaluated on the changed slice only. If the canary’s error budget burn exceeds the threshold while the baseline is healthy, roll back automatically. This avoids halting all releases due to ambient noise and makes the guardrail sensitive to the change under test.
It is common to start with success-rate SLOs at the edge because they correlate with real user pain and are available across ingress controllers and meshes. Latency SLOs complement them once request distributions are well understood. For latency, prefer upper percentiles over means, and only gate on percentiles that are stable for your traffic profile. Percentiles amplify cardinality and sample size issues; use recording rules to precompute percentiles and downsample with care. When distributions are spiky, pair a stricter fast window with a more tolerant long window to avoid false positives during brief surges.
Not all useful platform SLOs are request-centric. Workload readiness and scheduling SLOs protect tenants from systemic capacity or control plane problems. A simple objective such as “pods become Ready within 2 minutes of scheduling, 99% of the time” highlights autoscaler or image registry issues before application rollouts amplify them. The following rule derives a readiness SLI from pod conditions and creates a burn-rate alert that can freeze deploys when the platform is struggling to admit new work.
# PrometheusRule for pod readiness SLO measuring platform scheduling performance
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: platform-slo-readiness
namespace: observability
spec:
groups:
- name: platform.readiness.slo
interval: 30s
rules:
- record: slo:pod_readiness_within_2m:ratio_rate5m
expr: |
sum(rate(kube_pod_status_ready_time_seconds_bucket{le="120"}[5m]))
/
sum(rate(kube_pod_status_ready_time_seconds_count[5m]))
- record: slo:pod_readiness:error_budget
labels:
slo: "pod-readiness-2m"
objective: "99.0"
expr: 1 - 0.99
- record: slo:pod_readiness:burn_rate5m
expr: (1 - slo:pod_readiness_within_2m:ratio_rate5m) / slo:pod_readiness:error_budget
- alert: PlatformReadinessErrorBudgetBurn
expr: slo:pod_readiness:burn_rate5m > 10
for: 15m
labels:
severity: page
slo: "pod-readiness-2m"
guardrail: "release-freeze"
Guardrails should be enforced at every promotion boundary. A non-blocking pre-check in pull requests builds confidence without slowing developers. A blocking check in staging reduces bad promotions to production. A final gate before production deploys enforces the current platform health and the canary outcome. Keep gates deterministic: every check should reference a specific query, time range, and threshold. Store these definitions as code alongside the pipeline so that changes undergo review, and ensure the monitoring rules themselves are versioned and deployed through the same process as other cluster resources.
OpenTelemetry can simplify instrumentation across diverse workloads and avoid ad-hoc metric names. Standard semantic conventions let you aggregate success and latency across runtimes. The collector can scrape, transform, and forward metrics to Prometheus or remote storage, ensuring a single place to apply relabeling and tenancy tags. A minimal configuration that receives OTLP from applications, attaches environment labels, and remote-writes to Prometheus-compatible storage keeps pipelines straightforward.
# OpenTelemetry Collector configuration for metrics collection with environment labeling
receivers:
otlp:
protocols:
http:
grpc:
processors:
batch: {}
attributes:
actions:
- key: environment
value: "prod"
action: upsert
- key: cluster
value: "cluster-a"
action: upsert
exporters:
prometheusremotewrite:
endpoint: https://prom-remote.platform.local/api/v1/write
headers:
Authorization: Bearer ${PROM_REMOTE_TOKEN}
service:
pipelines:
metrics:
receivers: [otlp]
processors: [attributes, batch]
exporters: [prometheusremotewrite]
Multi-tenancy requires careful label hygiene. Every SLO query should be scoped to the environment and cluster at minimum. Namespace scoping is useful when you want to measure the platform’s effect on a particular tenant, but platform SLOs should not depend on any single team’s behavior. Avoid per-service cardinality explosions by aggregating at the edge and using standard labels for service and route. For canary gating, keep the version label stable and short. When labels or metric names change, treat that as a breaking change requiring updates to recording rules and pipeline gates.
Security is part of the guardrail design. The pipeline needs read-only access to the metrics API, not broad cluster credentials. Retrieve tokens or certificates from a secret manager and rotate them on a fixed cadence. Restrict who can modify SLO thresholds and which pipelines may bypass gates. When overrides are necessary, require a documented incident or change record and bound the time window of the override. Auditable release decisions build trust in the process and make post-incident analysis faster.
Cost is an unavoidable trade-off. Tightening SLOs usually implies more capacity or more conservative autoscaling, especially for tail latency. Before raising targets, model the expected impact on replicas, CPU headroom, and scale-up delays. Downsampling and recording rules control query load and storage costs, but they must not mask real regressions. Treat the SLO as a product constraint; renegotiate it when the economics change rather than letting overrides accumulate.
Operationally, treat guardrail failures as valuable signals, not blockers to be worked around. A failed pre-production gate is an early warning that catches a transient risk; rerun after the platform stabilizes. A failed production gate should pause promotions and trigger the listed runbook, which often includes checking concurrent incidents, validating data freshness, confirming that the SLI still matches intent, and, if necessary, reducing traffic to the canary or rolling back. Keep feedback loops short by surfacing the precise queries and recent metric samples directly in pipeline logs.
Common pitfalls include missing or lagging data, which leads to NaN results and false freezes; ensure gates treat missing data as a failure with a distinct exit code so engineers can fix telemetry rather than guessing. Percentile calculations suffer when histograms are poorly configured; align bucket boundaries with your SLOs. Label churn from automated versioning can bloat time series counts and slow queries; stabilize label values for long-lived series and limit high-cardinality dimensions such as user ID or request ID to logs or traces.
The practical path starts small. Choose one availability SLO at the edge and one readiness SLO for workloads. Instrument with recording rules, watch the dashboards for two weeks, and tune thresholds based on real distributions. Introduce a non-blocking gate in staging, then make it blocking, then extend to production. Add latency SLOs once you have confidence in histograms. Each step is simple, but the compound effect is meaningful: releases become a mechanical outcome of objective health signals rather than a subjective meeting about risk.
Platform SLOs anchor release decisions to measurable reality. By standardizing a few signals, computing error budgets with multi-window burn rates, and enforcing them as hard gates in CI/CD, Kubernetes platforms gain speed without sacrificing safety. The guardrails are transparent, reproducible, and easy to evolve as the platform and its tenants mature.
References
SRE Book - Service Level Objectives https://sre.google/sre-book/service-level-objectives/
Prometheus Documentation https://prometheus.io/docs/introduction/overview/
OpenTelemetry Documentation https://opentelemetry.io/docs/
Prometheus Operator https://prometheus-operator.dev/
SLO Best Practices https://sre.google/workbook/implementing-slos/