Large Language Models are good at summarizing incidents and navigating unfamiliar systems, but they rarely have first-class, governed access to production telemetry. The Model Context Protocol (MCP) changes that by standardizing how tools expose capabilities to LLMs. Recent additions in Grafana Cloud Traces and the upcoming Tempo 2.9 release bring an MCP server to tracing backends, allowing agents to query TraceQL safely with auditable policies. This article explains the architecture, security guardrails, and a reproducible path to pilot MCP-based access to traces in a multi-tenant environment.
The core pattern is straightforward: an MCP-compatible client (for example, an internal agent service) connects to a tracing MCP server that fronts your tracing backend. The server exposes a small set of tools mapping to TraceQL queries and trace retrieval endpoints. Because the server is a process you operate, it becomes the enforcement point for authentication, tenancy routing, and query shaping. Grafana Labs provides an MCP server for Grafana Cloud Traces and a Tempo-integrated server (merged for 2.9), which reduces the amount of glue code teams need to maintain.
MCP does not weaken access boundaries by default. The server sits out-of-band from your application traffic and uses the same transport and auth your tracing backend supports. For Cloud Traces, use the documented service tokens scoped to read-only trace APIs; on self-hosted Tempo, prefer reverse proxies that inject tenant headers and apply mTLS at the edge. Tempo’s multi-tenancy model is header-based and pairs well with MCP because a single MCP server can multiplex requests across tenants while emitting audit logs for the initiating principal. If you already enforce query RBAC in Tempo, reuse those policies so the MCP server cannot return sensitive spans or attributes outside approved scopes.
A minimal pilot starts with a non-production tenant and constrained data retention. Point the MCP server at a read replica or a dedicated query gateway to avoid contention with existing SLO dashboards. Keep the tool surface area small at first: “findTraces”, “getTrace”, “topServices”, and “topErrors” cover most operator workflows. Require explicit time windows and max result counts; reject unbounded scans. Layer rate limits per principal to prevent accidental cost explosions when agents iterate on prompts.
Operationally, the simplest deployment model is to run the MCP server as a stateless service in your platform cluster with a per-tenant configuration secret. Attach a NetworkPolicy or equivalent to restrict egress strictly to the tracing API endpoints. Emit structured audit logs from the MCP server itself: who invoked which tool, parameters, and query cost. Forward those logs to your SIEM and correlate with tracing backend access logs to validate coverage and incident response procedures.
For teams in Grafana Cloud, the path is shorter because the MCP endpoint is provided by the service. Provision a read token; limit the token to the traces datasource; and register the MCP endpoint with your chosen client. The benefit is immediacy and lower toil; the trade-off is fewer knobs for custom logic and on-path policy injection. For teams on self-hosted Tempo, the provided Tempo MCP server offers parity with Cloud Traces while allowing deeper customization at the cost of owning lifecycle, scaling, and upgrades. Track the Tempo 2.9 release notes for the merged MCP work and associated configuration switches.
A good first exercise is to use an MCP client to drive TraceQL queries that mirror the queries your team already runs in Grafana UI. Keep the query set curated and readable so that prompts can reference them by name. Bind each query to a tool with typed parameters such as service.name, span.kind, error flag, and time window. This nudges agents to request safe, bounded queries and gives you deterministic behavior for incident runbooks.
The following example shows a Kubernetes deployment of an MCP server that targets a Tempo query gateway for a single tenant. It relies on a secret that stores the target URL and a bearer token with read-only permissions. Replace domains and namespaces with your environment values and consider adding PodSecurity admission in production.
# Kubernetes deployment for Tempo MCP server with secure configuration and networking
apiVersion: v1
kind: Secret
metadata:
name: tempo-mcp-config
namespace: observability
type: Opaque
stringData:
TEMPO_URL: "https://tempo.qry.gw.example.com"
TEMPO_TENANT: "sandbox-tenant-01"
TEMPO_BEARER_TOKEN: "redacted-read-token"
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: tempo-mcp-server
namespace: observability
spec:
replicas: 2
selector:
matchLabels:
app: tempo-mcp-server
template:
metadata:
labels:
app: tempo-mcp-server
spec:
containers:
- name: server
image: ghcr.io/grafana/tempo-mcp-server:latest
envFrom:
- secretRef:
name: tempo-mcp-config
ports:
- containerPort: 8080
args:
- "--listen=0.0.0.0:8080"
- "--tenant=$(TEMPO_TENANT)"
readinessProbe:
httpGet:
path: /health
port: 8080
---
apiVersion: v1
kind: Service
metadata:
name: tempo-mcp
namespace: observability
spec:
selector:
app: tempo-mcp-server
ports:
- port: 8080
targetPort: 8080
name: http
Clients need a configuration that declares the MCP server endpoint and transports. The example below uses an SSE endpoint exposed via a cluster ingress, with a short timeout and a small tool catalog. Keep timeouts low during pilots to surface runaway queries early.
# MCP client configuration for connecting to Grafana Traces server via SSE transport
{
"mcpServers": {
"traces": {
"transport": {
"type": "sse",
"url": "https://mcp-traces.example.com/mcp"
},
"capabilities": ["resources", "tools"],
"metadata": {
"owner": "platform-observability",
"env": "nonprod"
}
}
}
}
Before enabling production tenants, define tenancy routing and headers explicitly. With Tempo, route per tenant using the standard header and restrict which tenants an MCP instance may reach. At the proxy layer, pin the header value and drop any client-supplied versions to prevent confused deputy problems.
# Envoy proxy configuration enforcing tenant isolation and rate limiting for Tempo MCP access
# (representative; adapt to your proxy)
route_config:
name: tempo
virtual_hosts:
- name: tempo
domains: ["tempo.qry.gw.example.com"]
routes:
- match: { prefix: "/api" }
request_headers_to_add:
- header:
key: "X-Scope-OrgID"
value: "sandbox-tenant-01"
append_action: APPEND_IF_EXISTS_OR_ADD
typed_per_filter_config:
envoy.filters.http.ratelimit:
"@type": type.googleapis.com/envoy.extensions.filters.http.ratelimit.v3.RateLimit
domain: "tempo_mcp"
stage: 0
timeout: "0.2s"
TraceQL queries invoked through MCP should be bounded and purposeful. Start with scoped selectors and visible limits. Validate each query server-side and reject patterns that expand cardinality without clear value. The following curl examples mirror two safe operations: finding recent error traces for a service and fetching a specific trace.
# TraceQL queries for service error investigation and trace retrieval via Tempo API
curl -s -H "Authorization: Bearer $TOKEN" \
-H "X-Scope-OrgID: sandbox-tenant-01" \
--data-urlencode 'query={ service.name = "payments" && status = error } | limit 20' \
--data-urlencode 'start=now()-15m' \
--data-urlencode 'end=now()' \
https://tempo.qry.gw.example.com/api/search
# Retrieve a trace by ID and pretty-print a summary with jq
curl -s -H "Authorization: Bearer $TOKEN" \
-H "X-Scope-OrgID: sandbox-tenant-01" \
https://tempo.qry.gw.example.com/api/traces/1f0c8e54a7c3f2b9 \
| jq '{traceId: .traceID, spans: (.spans | length), rootService: .spans[0].process.serviceName}'
Security posture hinges on keeping the MCP tool catalog minimal and mapping every call to deterministic backend requests. Avoid tools that accept free-form SQL-like predicates in the first iteration. Prefer enumerations and typed filters. Use allowlists for attribute keys and strip PII-bearing fields from results server-side. Attach rate limits to user identities rather than to service accounts so bursty agent behavior is attributable and controllable. Log request costs (rows scanned, spans returned) to detect regressions in prompt chains.
Governance requires treating the MCP server like any other production-facing API. Assign ownership, SLOs for availability and latency, and a standard release pipeline. Version tool schemas to avoid breaking agent integrations. Add smoke tests that run canonical TraceQL queries on every deploy and fail fast if the backend changes. Keep rollouts small and reversible; the cost of returning too much data is higher than returning too little.
From a cost perspective, MCP helps align LLM usage with observability spend. By pushing summarization closer to the data and constraining queries, you reduce round trips and payload sizes. However, summarization still consumes tokens; prefer structured results and let downstream prompts decide when to condense text. Cache recent query responses for short windows to absorb agent retries and minor prompt edits without re-querying the backend.
As features evolve, track the official documentation and release notes. Grafana Cloud Traces documents the MCP server endpoint and available tools, and the Tempo repository and notes identify when MCP features land in open source (2.9). For deeper customization, the published Tempo MCP server implementation is a good reference for extending tool definitions or adapting transports. These upstream references make it easier to maintain parity as your platform standardizes on MCP-based access to telemetry.
The practical takeaway is that MCP unlocks controlled, testable LLM access to your tracing data without bypassing existing boundaries. Start with a non-production tenant, a narrow tool set, and strict query shapes; prove you can answer real operational questions faster and more safely. When you are ready, expand to production with the same controls, knowing that both Grafana Cloud Traces and Tempo 2.9 provide a supported path for MCP in modern tracing stacks.
References
Grafana Tempo Documentation https://grafana.com/docs/tempo/latest/
Model Context Protocol Specification https://modelcontextprotocol.io/
TraceQL Query Language https://grafana.com/docs/tempo/latest/traceql/
Grafana Cloud Traces https://grafana.com/products/cloud/traces/
OpenTelemetry Tracing https://opentelemetry.io/docs/concepts/signals/traces/