AgentOps brings the discipline of microservice operations to AI agents: repeatable deployments, clear guardrails, and measurable outcomes. Kagent is an open‑source AI agent platform purpose‑built for Kubernetes. It provides AI‑powered automation, multi‑provider support (OpenAI, Anthropic, Azure OpenAI, Ollama, Gemini, etc.), tool integration, agent‑to‑agent communication, comprehensive observability and cloud‑native deployment. Rather than chasing novelty, the goal of AgentOps is reliable automation that can be promoted through environments without surprises.
The core idea is to treat agents as first‑class workloads that can be declared, observed and constrained. Kubernetes supplies the isolation, policy and rollout machinery; kagent stitches these together with a controller, engine, CLI and dashboard. Kagent supports tools via the Model Context Protocol (MCP) but you can also use built‑in Kubernetes tools or standard HTTP endpoints. This post outlines a practical AgentOps baseline using kagent, focusing on kagent features rather than the underlying MCP details.
Install kagent and its CLI in your cluster. The CLI download script installs a kagent binary, and the kagent install command installs both the kagent controller and CRDs; the kmcp subchart is now included, so you no longer need to install separate kmcp CRDs. Replace the OpenAI API key with credentials appropriate for your environment.
# Install kagent CLI and bootstrap development environment
set -euo pipefail
export OPENAI_API_KEY="redacted-dev-key"
# Download the kagent CLI (installs to ~/.local/bin)
curl -fsSL https://raw.githubusercontent.com/kagent-dev/kagent/refs/heads/main/scripts/get-kagent | bash
# Install kagent CRDs and components (includes kmcp subchart)
kagent install
# Open the dashboard (port‑forwarded locally)
kagent dashboard
The kagent dashboard command port‑forwards the UI service and opens the browser. You can explore agents, tools and conversations from this dashboard.
Once kagent is running, start with a minimal operator feedback loop in the terminal. Kagent's CLI exposes commands such as get and invoke. These are useful for smoke tests in CI and for verifying that an agent still works after promotion.
# List available agents and test basic kagent functionality
set -euo pipefail
kagent get agent
# Invoke an agent non‑interactively (capture output and exit code in CI)
kagent invoke --agent helm-agent -t "List deployed Helm charts across all namespaces"
The get subcommand lists Agent CRs in the cluster, while invoke triggers a conversation with the specified agent and prints the tool calls and responses.
Kagent separates model configuration from agent behaviour. Use a Kubernetes Secret to store credentials and a ModelConfig custom resource to describe the model and provider. This keeps agent definitions declarative and portable, while allowing platform teams to rotate keys or switch providers without touching agent specs. For OpenAI models, kagent automatically configures capabilities like function‑calling and JSON output.
Create a secret and a ModelConfig resource:
# Create Kubernetes Secret containing your OpenAI API key
set -euo pipefail
kubectl -n kagent create secret generic kagent-openai \
--from-literal=OPENAI_API_KEY="$OPENAI_API_KEY"
# ModelConfig resource defining an OpenAI model and authentication
apiVersion: kagent.dev/v1alpha2
kind: ModelConfig
metadata:
name: custom-openai-model-config
namespace: kagent
spec:
apiKeySecret: kagent-openai
apiKeySecretKey: OPENAI_API_KEY
model: gpt-4o-mini
Tools are the agent's way of acting in the real world. Use MCP for clean, inspectable interfaces and lifecycle separation. The pattern is to register an MCP server behind a Service, label it for discovery, and constrain which tools an agent can access. This keeps different agents from sharing broad, implicit capabilities and makes reviews straightforward.
# MCP server Service configuration for website fetching tools with discovery labels
apiVersion: v1
kind: Service
metadata:
name: mcp-website-fetcher
namespace: kagent
labels:
kagent.dev/mcp-service: "true"
annotations:
kagent.dev/mcp-service-path: /mcp
kagent.dev/mcp-service-port: "8000"
kagent.dev/mcp-service-protocol: STREAMABLE_HTTP
spec:
selector:
app: mcp-website-fetcher
ports:
- port: 80
targetPort: 8000
type: ClusterIP
# Declarative Agent configuration with restricted tool access via MCP server
apiVersion: kagent.dev/v1alpha2
kind: Agent
metadata:
name: simple-fetch-agent
namespace: kagent
spec:
type: Declarative
declarative:
modelConfig: custom-openai-model-config
systemMessage: |
You retrieve webpage content using the fetch tool and summarize risks.
Ask clarifying questions if the request is ambiguous.
tools:
- type: McpServer
mcpServer:
kind: Service
name: mcp-website-fetcher
toolNames:
- fetch
Least‑privilege RBAC is the most important blast‑radius control for operational agents. Create a dedicated ServiceAccount per agent, bind a minimal Role, and use that ServiceAccount in the agent's workload. The Role should only authorize read actions that the agent actually uses. Write access, if ever necessary, must be scoped to a narrow set of resources and always paired with change‑approval controls outside the agent loop.
# RBAC configuration granting minimal read-only permissions to agent ServiceAccount
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: agent-readonly
namespace: kagent
rules:
- apiGroups: [""]
resources: ["pods", "events", "services", "endpoints"]
verbs: ["get", "list", "watch"]
- apiGroups: ["apps"]
resources: ["deployments", "replicasets", "statefulsets"]
verbs: ["get", "list", "watch"]
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: simple-agent-sa
namespace: kagent
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: agent-readonly-binding
namespace: kagent
subjects:
- kind: ServiceAccount
name: simple-agent-sa
roleRef:
kind: Role
name: agent-readonly
apiGroup: rbac.authorization.k8s.io
Agents should be cost‑aware and predictable. Use a LimitRange to set default CPU and memory requests and hard limits, and a ResourceQuota to cap aggregate consumption. Pair these with reasonable timeouts and tool‑level concurrency limits.
# LimitRange and ResourceQuota to control resource consumption
apiVersion: v1
kind: LimitRange
metadata:
name: agent-limits
namespace: kagent
spec:
limits:
- type: Container
default:
cpu: "500m"
memory: "512Mi"
defaultRequest:
cpu: "100m"
memory: "128Mi"
---
apiVersion: v1
kind: ResourceQuota
metadata:
name: agent-quota
namespace: kagent
spec:
hard:
pods: "20"
requests.cpu: "4"
requests.memory: "8Gi"
limits.cpu: "8"
limits.memory: "16Gi"
Network controls keep agents honest when they call out to external systems. Deny‑by‑default with explicit egress allows you to whitelist model gateways, internal APIs and package registries. Add DNS and API server exceptions as needed. In multi‑tenant clusters, pair this with Namespaces per agent or per team and prohibit cross‑namespace traffic with additional policies.
# NetworkPolicy restricting agent egress to essential services only
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: agent-egress-guardrails
namespace: kagent
spec:
podSelector: {}
policyTypes: ["Egress"]
egress:
- to:
- namespaceSelector: {}
ports:
- port: 53
protocol: UDP
- to:
- ipBlock:
cidr: 10.0.0.0/8 # Kubernetes API server or internal VIPs
ports:
- port: 443
protocol: TCP
- to:
- namespaceSelector: {}
podSelector:
matchLabels:
app: model-gateway
ports:
- port: 443
protocol: TCP
Observability is the bedrock of AgentOps. Instrumentation should capture prompts, tool calls, result summaries, latency and provider‑side usage metrics. Kagent exposes OpenTelemetry traces and metrics, and its key‑features page highlights built‑in tracing and monitoring. Ensure correlation identifiers propagate through tool calls and logs, and route structured logs to a tamper‑evident store. Set alerts on budget thresholds and concurrency caps.
When promoting agents through environments, use pre‑production sandboxes with constrained providers and synthetic data. In production, start with narrow scopes and a "two‑key" flow for mutating changes, where an agent proposes a manifest and a separate controller applies it after policy evaluation. Keep all agent specifications and tool whitelists under version control so changes can be reviewed like any other operational change.
Cost and performance are intertwined. Set temperature and concurrency values deliberately; avoid unnecessary tool invocations; cache immutable results behind MCP servers. Use short‑lived agents for bursty tasks and long‑lived agents only when their stateful context is valuable. Maintain per‑namespace budgets tied to provider usage.
Resilience requires pessimism about dependencies. Assume model gateways will throttle, MCP servers will crash and tools will return malformed output. Use circuit breakers and exponential backoff in MCP servers, idempotent tool implementations and strict schemas for inputs/outputs. Prefer "read‑verify‑write" for any mutating action and ensure you can replay sessions deterministically from logs and traces.
These patterns do not mandate kagent, but kagent's declarative resources, CLI and dashboard make them straightforward to implement. By treating agents as first‑class workloads, separating model configuration, constraining tool access and enforcing least privilege, you can build a pragmatic AgentOps stack on Kubernetes: traceable, budget‑aware automation that promotes with confidence. Start small, keep the guardrails tight, and expand scope as you gain evidence that your agents deliver value under real operational constraints.
References
kagent Documentation https://kagent.dev/
kagent GitHub Repository https://github.com/kagent-dev/kagent
Kubernetes RBAC Reference https://kubernetes.io/docs/reference/access-authn-authz/rbac/
Kubernetes NetworkPolicy Guide https://kubernetes.io/docs/concepts/services-networking/network-policies/
OpenTelemetry Kubernetes Observability https://opentelemetry.io/docs/kubernetes/getting-started/
Kubernetes Resource Management Guide https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/