Selected work

Agentic AI · Kubernetes

Ongoing

Autonomous AIOps Agent

Project purpose

An AI-powered Kubernetes diagnostic system that uses LangGraph to investigate infrastructure failures through controlled, read-only tool calls.

  • LangGraph
  • FastAPI
  • Kubernetes
  • Python
  • RBAC

Offline tests

127 passed in the audited local run

Access controls

API namespace authorization and Kubernetes RBAC

Kind validation

Live collector test on a failing lab workload

The engineering problem

A Pod’s status can tell an operator that a container is restarting. The reason may be in the previous container’s logs, a Kubernetes Event, or a resource limit. Investigating by hand means moving between APIs and matching records to the same Pod and container. The engineering challenge is to bring those observations together while keeping a suspected cause distinct from a confirmed symptom.

Conceptual flow

System architecture

FastAPI authenticates the operator and authorizes namespace access before LangGraph runs. Diagnostic tools use ServiceAccount credentials for read-only Kubernetes requests, where RBAC is enforced independently. Evidence returns through ToolNode to AgentNode. A diagnosis returns to the operator through FastAPI.

Conceptual execution sequence

  1. Investigation request
  2. Bearer authentication and namespace authorization
  3. LangGraph selects and dispatches a diagnostic tool
  4. The Kubernetes API authenticates the ServiceAccount and enforces RBAC
  5. Pod status, logs, Events and available metrics return to the collector
  6. Evidence returns to AgentNode, which can request another tool call
  7. FastAPI returns the diagnosis to the operator

Captured Kind lab evidence

Investigating CrashLoopBackOff

The diagnostic CLI collected this snapshot from aiops/crash-demo, a deliberately failing Pod. The request below is an example. The interpretation comes from reviewing the manifest and collector output.

Captured . No agent-selected tool trace or model answer was captured.

Observed Pod state

CrashLoopBackOff

Pod phase
Running
Container ready
No
Previous exit code
1
Restart count at capture
2,874

Previous container logs

Starting application...
ERROR: Database connection failed

Kubernetes Events

  • NormalPulled51 reported
  • WarningBackOff216 reported

Pod status, bounded current and previous logs, and events were collected. The metrics API returned HTTP 404.

Human interpretation

The Pod phase is Running while its container is waiting in CrashLoopBackOff. The previous exit code was 1, and Kubernetes recorded BackOff. The fixture prints the database error and unconditionally exits. The log line alone does not establish a database outage.

Example investigation request

Requires an authenticated account with access to the requested namespace.

POST /agent/investigate

{
  "namespace": "aiops",
  "message": "Investigate Pod crash-demo in namespace aiops. Use Pod status, previous container logs, and Kubernetes Events. Note whether metrics are available and distinguish observations from possible causes."
}
Collectors executed during the capture

The diagnostic CLI orchestrated collectors directly. The agent adapters are listed for reference.

collect_pod_information
get_pod_info
collect_container_logs
get_logs
collect_kubernetes_events
get_events
collect_resource_metrics
get_metrics

Engineering decisions

Two authorization boundaries

FastAPI checks the operator’s permitted namespaces before collection. Kubernetes independently limits the ServiceAccount to specific read verbs through a RoleBinding. The two checks protect different identities and are configured separately.

Bounded evidence

Current and previous logs use both a line limit and a per-stream byte limit. Events and Pod listing are bounded, and missing metrics remain an explicit collection error. The caller can use partial evidence without treating a missing observation as proof.

A tool loop with a step limit

LangGraph routes model tool calls to ToolNode, appends their results to message state, and permits another tool call or a final answer. The graph step limit prevents repeated calls from running indefinitely. It can also cut short a difficult investigation.

Correlatable failures

The response pairs an investigation ID with the answer and counts derived from structured tool messages. The ID joins private operations and message logs. An error late in the graph retains earlier evidence, and the HTTP response uses safe failure text.

Validation

Offline automated tests

127 tests passed in the audited local run. They cover graph routing, API authorization, error responses, log redaction, and Helm RBAC templates with mocked model and cluster dependencies. GitHub Actions is configured to run offline checks.

Kind integration

The separately run, opt-in test_crash_demo test passed against the local lab. It confirms that the restricted collector returns a serializable report for the failing Pod, including its container logs.

Security verification

Live access reviews allowed the lab ServiceAccount to read Pod logs in aiops. Kubernetes denied Pod deletion there and Pod listing in default. No deletion was attempted.

Technology stack

AI
LangGraph, LangChain
Backend
Python, FastAPI
Infrastructure
Kubernetes, Kind, Helm
Security
API authorization, Kubernetes RBAC
Engineering
Python unittest, GitHub Actions

Implementation references