AIOps in Practice: Using LLMs for Log Analysis, Incident Correlation, and Automated Root Cause Analysis

Traditional observability stacks are good at showing you what happened. They are poor at explaining why. A well-tuned Prometheus setup will fire alerts when p99 latency crosses a threshold. Grafana will show you the spike. What neither will do is tell you that the latency spike correlates with a slow database migration started by a background job, which was triggered by a feature flag rollout three minutes earlier, on a service two hops away.

That gap is where LLMs can actually add value. Not as a replacement for metrics pipelines or alerting rules, but as a reasoning layer over structured telemetry that is expensive or impossible to encode as static rules.

This article covers four concrete applications: clustering similar log errors with embeddings, correlating incidents across services with LLM reasoning, building automated root cause analysis pipelines, and reducing alert noise. For each, the honest answer about where LLMs help and where they do not.

Log Pattern Extraction with Embeddings

The first problem is clustering. In a microservices environment, the same underlying error manifests in dozens of slightly different log lines. connection refused: 10.0.0.5:5432, ECONNREFUSED connecting to postgres, dial tcp: connect: connection refused are all the same event. A rule that matches one misses the others. Regex maintenance across twenty services is a full-time job.

Embedding-based clustering handles this because semantically similar log lines produce nearby vectors, regardless of the exact template.

import OpenAI from "openai";
import { kmeans } from "ml-kmeans";

const openai = new OpenAI();

interface LogEntry {
  id: string;
  timestamp: string;
  service: string;
  message: string;
  level: string;
}

interface LogCluster {
  centroidLabel: string;
  members: LogEntry[];
  representativeMessage: string;
}

async function embedLogs(logs: LogEntry[]): Promise<number[][]> {
  // Strip dynamic tokens before embedding: IPs, UUIDs, timestamps, numeric IDs
  const normalized = logs.map((log) =>
    log.message
      .replace(/\b\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}(:\d+)?\b/g, "<IP>")
      .replace(/[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}/gi, "<UUID>")
      .replace(/\b\d{4,}\b/g, "<NUM>")
  );

  const response = await openai.embeddings.create({
    model: "text-embedding-3-small",
    input: normalized,
  });

  return response.data.map((d) => d.embedding);
}

async function clusterLogErrors(
  logs: LogEntry[],
  k: number = 8
): Promise<LogCluster[]> {
  const embeddings = await embedLogs(logs);

  // k-means over the embedding space
  const result = kmeans(embeddings, k, { initialization: "kmeans++" });

  const clusters: Map<number, LogEntry[]> = new Map();
  result.clusters.forEach((clusterIndex, logIndex) => {
    const existing = clusters.get(clusterIndex) ?? [];
    existing.push(logs[logIndex]);
    clusters.set(clusterIndex, existing);
  });

  // Label each cluster with a short description using the LLM
  const labeled: LogCluster[] = [];
  for (const [, members] of clusters) {
    const sample = members.slice(0, 5).map((m) => m.message).join("\n");
    const labelResponse = await openai.chat.completions.create({
      model: "gpt-4o-mini",
      messages: [
        {
          role: "user",
          content: `Summarize the following log messages as a short error pattern label (max 8 words):\n\n${sample}`,
        },
      ],
      max_tokens: 30,
    });

    labeled.push({
      centroidLabel: labelResponse.choices[0].message.content?.trim() ?? "unknown",
      members,
      representativeMessage: members[0].message,
    });
  }

  return labeled;
}

The normalization step before embedding is critical. Without stripping UUIDs and IPs, two identical errors on different hosts produce different vectors. The embedding model treats the IP as meaningful content.

One practical concern: k is a free parameter and you do not always know the right number of clusters in advance. HDBSCAN avoids this by finding clusters based on density, at the cost of more infrastructure to run. For a weekly batch job over a day’s worth of error logs, k-means with a reasonable k (start at 10-15 for medium-size services) is sufficient. For real-time streaming clustering, the maintenance cost of HDBSCAN is justified.

Incident Correlation Across Services

Distributed traces give you the causal chain within a single request. What they do not give you is correlation across requests that are causally related but did not share a trace. A slow third-party API call is not in your trace. A database lock held by a background job is not in the request trace for the user who timed out waiting.

This is where LLM reasoning over structured telemetry is genuinely useful. The model does not need to understand your codebase. It needs a structured summary of what changed near a given timestamp and the ability to reason about plausible causal relationships.

interface ServiceEvent {
  service: string;
  timestamp: string;
  eventType: "deployment" | "config-change" | "spike" | "error-rate-increase" | "latency-p99-spike";
  description: string;
  traceId?: string;
}

interface IncidentContext {
  incidentId: string;
  affectedService: string;
  startTime: string;
  windowMinutes: number;
  events: ServiceEvent[];
  topErrorClusters: string[];
}

async function correlateIncident(ctx: IncidentContext): Promise<string> {
  const eventSummary = ctx.events
    .sort((a, b) => a.timestamp.localeCompare(b.timestamp))
    .map(
      (e) =>
        `[${e.timestamp}] ${e.service} — ${e.eventType}: ${e.description}${e.traceId ? ` (trace: ${e.traceId})` : ""}`
    )
    .join("\n");

  const prompt = `
You are analyzing a production incident on service "${ctx.affectedService}" that started at ${ctx.startTime}.

Events observed in all services in the ${ctx.windowMinutes} minutes before and after the incident:

${eventSummary}

Top error clusters on the affected service:
${ctx.topErrorClusters.map((c, i) => `${i + 1}. ${c}`).join("\n")}

Using chain-of-thought reasoning:
1. Identify which events preceded the incident onset
2. Assess whether any upstream service change could have cascaded
3. State your most likely root cause hypothesis with confidence (low/medium/high)
4. List what additional data would confirm or refute it

Be concise. Flag uncertainty explicitly. Do not speculate beyond the evidence provided.
`;

  const response = await openai.chat.completions.create({
    model: "gpt-4o",
    messages: [{ role: "user", content: prompt }],
    temperature: 0,
    max_tokens: 600,
  });

  return response.choices[0].message.content ?? "";
}

The temperature: 0 setting is not optional for this use case. Incident analysis needs deterministic output so you can re-run the same context and get comparable results. A non-zero temperature produces different hypotheses on different runs over identical data, which makes it impossible to build confidence in the system’s output over time.

The data feeding this function matters more than the prompt. You need a pipeline that, at incident trigger time, fetches: deployment events from your CI/CD system, config change events from your feature flag store, service-level metrics summaries (p99 latency, error rate deltas), and the top error clusters from the affected service in the incident window. Assembling this context reliably is 80% of the engineering work.

Automated Root Cause Analysis Pipeline

Individual incident correlation is useful. A pipeline that runs it continuously, persists results, and surfaces patterns across incidents is what earns the “automated” label.

interface RCAResult {
  incidentId: string;
  timestamp: string;
  hypothesis: string;
  confidence: "low" | "medium" | "high";
  contributingEvents: string[];
  dataGaps: string[];
  resolvedAt?: string;
  actualRootCause?: string;
  hypothesisCorrect?: boolean;
}

async function runRCAPipeline(
  incidentId: string,
  telemetryFetcher: (incidentId: string) => Promise<IncidentContext>
): Promise<RCAResult> {
  const ctx = await telemetryFetcher(incidentId);
  const analysis = await correlateIncident(ctx);

  // Parse structured fields from the LLM response
  // In production, prompt the LLM to return structured JSON instead
  const confidenceMatch = analysis.match(/confidence[:\s]+(low|medium|high)/i);
  const confidence = (confidenceMatch?.[1]?.toLowerCase() ?? "low") as RCAResult["confidence"];

  return {
    incidentId,
    timestamp: new Date().toISOString(),
    hypothesis: analysis,
    confidence,
    contributingEvents: ctx.events.map((e) => `${e.service}: ${e.eventType}`),
    dataGaps: [],
  };
}

// Close the feedback loop: when an incident is resolved, record the actual root cause
async function recordResolution(
  rcaStore: Map<string, RCAResult>,
  incidentId: string,
  actualRootCause: string
): Promise<void> {
  const existing = rcaStore.get(incidentId);
  if (!existing) return;

  const updated: RCAResult = {
    ...existing,
    resolvedAt: new Date().toISOString(),
    actualRootCause,
    // Simple heuristic: check if the actual cause appears in the hypothesis text
    hypothesisCorrect: existing.hypothesis
      .toLowerCase()
      .includes(actualRootCause.toLowerCase().split(" ")[0]),
  };

  rcaStore.set(incidentId, updated);

  // Log accuracy metrics for the prompt improvement loop
  console.log(
    JSON.stringify({
      incidentId,
      confidence: existing.confidence,
      correct: updated.hypothesisCorrect,
    })
  );
}

The feedback loop at the end is what makes this a system rather than a script. Without tracking hypothesisCorrect over time, you cannot measure whether the LLM is actually helping or just producing plausible-sounding noise. In practice, you want an accuracy dashboard: what is the hit rate at each confidence level? If “medium” confidence hypotheses are correct 60% of the time and “high” confidence is correct 55% of the time, your confidence calibration is broken and needs work.

Structured JSON output from the LLM is strongly preferred over parsing free text. Use a response format schema:

const response = await openai.chat.completions.create({
  model: "gpt-4o",
  messages: [{ role: "user", content: prompt }],
  response_format: {
    type: "json_schema",
    json_schema: {
      name: "rca_analysis",
      schema: {
        type: "object",
        properties: {
          primaryHypothesis: { type: "string" },
          confidence: { type: "string", enum: ["low", "medium", "high"] },
          contributingFactors: {
            type: "array",
            items: { type: "string" },
          },
          dataGaps: {
            type: "array",
            items: { type: "string" },
          },
        },
        required: ["primaryHypothesis", "confidence", "contributingFactors", "dataGaps"],
      },
    },
  },
  temperature: 0,
});

Alert Noise Reduction

Alert fatigue is real. Most teams have monitors that fire correctly but not usefully: a downstream error rate alert fires because an upstream service went down, so you get 40 alerts for one incident. An LLM-based deduplication layer can group correlated alerts before they hit the on-call channel.

interface Alert {
  id: string;
  service: string;
  timestamp: string;
  severity: "critical" | "warning" | "info";
  message: string;
  labels: Record<string, string>;
}

interface AlertGroup {
  leadAlert: Alert;
  correlatedAlerts: Alert[];
  summary: string;
  likelyCause: string;
}

async function groupAlerts(
  alerts: Alert[],
  windowMinutes: number = 5
): Promise<AlertGroup[]> {
  if (alerts.length === 0) return [];

  // Sort by timestamp and filter to window
  const windowStart = Date.now() - windowMinutes * 60 * 1000;
  const recent = alerts.filter(
    (a) => new Date(a.timestamp).getTime() > windowStart
  );

  if (recent.length <= 1) {
    return recent.map((a) => ({
      leadAlert: a,
      correlatedAlerts: [],
      summary: a.message,
      likelyCause: "unknown",
    }));
  }

  const alertSummary = recent
    .map(
      (a) =>
        `[${a.timestamp}] ${a.service} (${a.severity}): ${a.message}`
    )
    .join("\n");

  const response = await openai.chat.completions.create({
    model: "gpt-4o-mini",
    messages: [
      {
        role: "user",
        content: `Group these alerts by likely common cause. Return JSON with an array of groups, each with: leadAlertIndex (index of the most informative alert), correlatedAlertIndexes (array of related alert indexes), summary (one sentence), likelyCause (one sentence).\n\nAlerts:\n${alertSummary}`,
      },
    ],
    response_format: { type: "json_object" },
    temperature: 0,
    max_tokens: 400,
  });

  const parsed = JSON.parse(response.choices[0].message.content ?? "{}");
  const groups: AlertGroup[] = (parsed.groups ?? []).map(
    (g: { leadAlertIndex: number; correlatedAlertIndexes: number[]; summary: string; likelyCause: string }) => ({
      leadAlert: recent[g.leadAlertIndex],
      correlatedAlerts: (g.correlatedAlertIndexes ?? []).map(
        (i: number) => recent[i]
      ),
      summary: g.summary,
      likelyCause: g.likelyCause,
    })
  );

  return groups;
}

gpt-4o-mini is the right model here. Alert grouping does not require deep reasoning. The context window is small (a batch of recent alerts). The latency budget is tight (you want this before the pager fires). At roughly 1/20th the cost of gpt-4o, mini handles this job well.

Cost and Latency Tradeoffs

This is where honest assessment matters most. Running LLMs in the observability hot path, meaning synchronously during incident detection, is expensive and adds latency. Here are the actual tradeoffs:

Approach	Latency	Cost per incident	Accuracy	When to use
Real-time RCA on every alert	2-8s added	$0.05-$0.20	Good	Only for critical severity, small volume
Batch RCA after alert fires	30-60s delay	$0.02-$0.10	Good	Most production use cases
Background log clustering (hourly)	Minutes	$0.50-$2.00/day	High (more context)	Error pattern detection, trend analysis
Alert grouping (5-min window)	1-3s	$0.001-$0.005/batch	Good	Always reasonable for noise reduction
Post-incident analysis (async)	N/A (async)	$0.10-$0.50	Highest	Weekly summaries, pattern mining

The practical recommendation: run alert grouping synchronously (latency is low, cost is negligible, value is high). Run incident RCA in batch mode 30-60 seconds after alert onset, not inline with the alert pipeline. Run log clustering as a scheduled job, not in a streaming pipeline. Save real-time synchronous LLM calls for the highest-severity events where the cost and latency are justified by the time saved in manual triage.

Token costs also compound. An incident context with 7 days of deployment history, 500 recent log lines, and all active alerts can easily hit 4,000-6,000 input tokens. At scale, context trimming is not optional. The code in this article sends 10 representative log lines per cluster rather than all raw logs. This is a real engineering tradeoff: you lose signal on rare edge cases, but you keep cost linear rather than exponential as log volume grows.

Where LLMs Win vs. Where Rules Still Win

LLMs are good at: reasoning over heterogeneous, unstructured text; identifying plausible causal chains across services; generating human-readable explanations; clustering semantically similar events without predefined templates; and surfacing non-obvious correlations in small-to-medium context windows.

Rules are better at: sub-millisecond detection (no LLM fits here); high-volume streaming pipelines where cost compounds fast; situations where the rule is known and stable (a 5xx spike above threshold should always page); compliance and audit scenarios where the detection logic must be auditable and deterministic; and any context where false negatives are catastrophic (security alerts should not be “denoised” by an LLM).

The practical dividing line: if you can write the rule and the rule is stable, write the rule. Use LLMs for the cases where you cannot enumerate the patterns in advance or where the value is in explaining rather than detecting.

Production Considerations

A few things that matter at production scale that do not show up in prototype code:

Context window management is a real constraint. As the number of services grows, the incident context window grows with it. At 20+ services, you need to be selective about what you include. Use recency weighting: events closer to the incident onset get higher priority. Use relevance filtering: only include services with dependency relationships to the affected service, which you can derive from your service mesh topology.

Prompt versioning matters. As you improve prompts, the RCA output format changes. If you are storing RCA results in a database and building dashboards over them, a prompt change that alters the JSON schema will break downstream consumers. Version your prompts and include the prompt version in the stored result.

Rate limits hit you during incidents. An outage that affects ten services generates ten simultaneous RCA triggers. If you have not built queue-based dispatch with retry logic, you will get throttled by the LLM API exactly when you need it most. Use a job queue with exponential backoff and priority levels.

Model upgrades invalidate embeddings. If you swap from text-embedding-ada-002 to text-embedding-3-small, every stored embedding is now in a different vector space. Your existing log clusters are meaningless. Plan for a re-embedding job when upgrading models, and maintain model version metadata alongside every stored embedding.

Closing Thought

LLMs in operations are useful when you treat them as a reasoning layer, not a detection layer. The telemetry pipelines, the metrics, the alerting thresholds: those stay. The LLM sits above them, reading structured summaries, surfacing hypotheses, and explaining correlations that would take a human 20 minutes to trace manually. That 20-minute reduction in mean time to understand is where the real value lives. The infrastructure to make it work reliably is unglamorous, but it is the part that determines whether the system earns trust with the team or gets turned off after the first wrong hypothesis.