AI FinOps for Engineering Teams: Cost Observability, Budget Enforcement, and Runaway Detection for LLM Infrastructure

Most teams discover they have an AI cost problem the same way: a Slack message from the CFO with a screenshot of the billing dashboard. The number is larger than anyone expected. Nobody can explain which feature, team, or agent produced it.

Per-request token optimization is a solved problem at this point. Caching, model routing, context compression: these are well-documented techniques. The harder problem is organizational. Who owns the bill? How do you attribute cost to a team when three features share the same OpenAI org? What happens when an agent enters a retry loop at 2 AM and burns $800 before anyone notices?

This article covers the governance layer that most teams build too late: team-level budget enforcement, real-time cost attribution, rogue agent detection, shadow AI spend, and zombie resource cleanup.

The Attribution Gap

The first problem is that AI provider billing gives you totals, not attribution. OpenAI’s usage dashboard shows you aggregate token counts by model. It does not show you which feature, which team, or which agent produced them. Anthropic’s console is similar. If you run multiple products or have multiple teams using the same provider account, you are flying blind.

The standard fix is to add a metadata layer in front of every LLM call. Every request gets tagged with a team identifier, a feature identifier, and an agent identifier when applicable. Those tags flow into a cost ledger that you control.

interface LLMCallMetadata {
  teamId: string;
  featureId: string;
  agentId?: string;
  userId?: string;
  traceId: string;
  environment: "production" | "staging" | "development";
}

interface CostLedgerEntry {
  id: string;
  timestamp: Date;
  metadata: LLMCallMetadata;
  model: string;
  provider: "openai" | "anthropic" | "google" | "together" | "bedrock";
  promptTokens: number;
  completionTokens: number;
  totalTokens: number;
  costUsd: number;
  latencyMs: number;
}

// Canonical pricing table (update as providers change rates)
const MODEL_PRICING_USD_PER_1M: Record<string, { input: number; output: number }> = {
  "gpt-4o": { input: 2.50, output: 10.00 },
  "gpt-4o-mini": { input: 0.15, output: 0.60 },
  "claude-3-5-sonnet-20241022": { input: 3.00, output: 15.00 },
  "claude-3-haiku-20240307": { input: 0.25, output: 1.25 },
  "gemini-1.5-flash": { input: 0.075, output: 0.30 },
  "meta-llama/Llama-3-70b-chat-hf": { input: 0.90, output: 0.90 },
};

function computeCost(
  model: string,
  promptTokens: number,
  completionTokens: number
): number {
  const pricing = MODEL_PRICING_USD_PER_1M[model];
  if (!pricing) return 0; // unknown model: flag for review
  return (
    (promptTokens / 1_000_000) * pricing.input +
    (completionTokens / 1_000_000) * pricing.output
  );
}

Store every entry in Postgres. The schema needs indexes on (teamId, timestamp) and (agentId, timestamp) to support the queries you will run daily.

Budget Enforcement Middleware

Attribution tells you what happened. Budget enforcement prevents runaway spend before it happens. The pattern is a middleware layer that checks a team’s remaining budget before dispatching an LLM call, and rejects or degrades the request if the budget is exhausted.

interface TeamBudget {
  teamId: string;
  periodStart: Date;
  periodEnd: Date;
  limitUsd: number;
  spentUsd: number;
  alertThresholds: number[]; // e.g. [0.5, 0.75, 0.9]
  hardStop: boolean; // if false: allow overage, alert; if true: reject at limit
}

interface BudgetCheckResult {
  allowed: boolean;
  remainingUsd: number;
  utilizationPct: number;
  degradeToModel?: string; // fallback if soft limit
}

async function checkBudget(
  teamId: string,
  estimatedCostUsd: number,
  db: DatabaseClient
): Promise<BudgetCheckResult> {
  const budget = await db.queryOne<TeamBudget>(
    `SELECT * FROM team_budgets
     WHERE team_id = $1
       AND period_start <= NOW()
       AND period_end > NOW()
     LIMIT 1`,
    [teamId]
  );

  if (!budget) {
    // No budget configured: allow but emit a warning metric
    return { allowed: true, remainingUsd: Infinity, utilizationPct: 0 };
  }

  const remainingUsd = budget.limitUsd - budget.spentUsd;
  const utilizationPct = budget.spentUsd / budget.limitUsd;

  if (budget.hardStop && estimatedCostUsd > remainingUsd) {
    return { allowed: false, remainingUsd, utilizationPct };
  }

  // Soft degradation: swap to a cheaper model as utilization rises
  let degradeToModel: string | undefined;
  if (utilizationPct >= 0.9 && !budget.hardStop) {
    degradeToModel = "gpt-4o-mini"; // cheapest capable fallback
  }

  return { allowed: true, remainingUsd, utilizationPct, degradeToModel };
}

// Token estimation before the actual call
function estimateTokens(prompt: string): number {
  // ~4 characters per token is a reasonable estimate for English text
  return Math.ceil(prompt.length / 4);
}

async function dispatchWithBudgetCheck(
  teamId: string,
  featureId: string,
  prompt: string,
  preferredModel: string,
  llmClient: LLMClient,
  db: DatabaseClient
): Promise<LLMResponse> {
  const estimatedPromptTokens = estimateTokens(prompt);
  // Assume completion is at most 2x prompt for estimation
  const estimatedCost = computeCost(preferredModel, estimatedPromptTokens, estimatedPromptTokens * 2);

  const budgetCheck = await checkBudget(teamId, estimatedCost, db);

  if (!budgetCheck.allowed) {
    throw new BudgetExceededError(
      `Team ${teamId} has exhausted its LLM budget for this period. ` +
      `Remaining: $${budgetCheck.remainingUsd.toFixed(4)}`
    );
  }

  const model = budgetCheck.degradeToModel ?? preferredModel;
  const response = await llmClient.complete({ model, prompt });

  // Record actual cost after the call
  const actualCost = computeCost(model, response.promptTokens, response.completionTokens);
  await db.execute(
    `UPDATE team_budgets
     SET spent_usd = spent_usd + $1
     WHERE team_id = $2 AND period_start <= NOW() AND period_end > NOW()`,
    [actualCost, teamId]
  );

  await recordLedgerEntry({ teamId, featureId, model, response, actualCost }, db);

  return response;
}

Two notes on this implementation. First, the UPDATE on spent_usd is non-atomic with the SELECT in checkBudget. Under concurrent load, two requests can both read a budget under the limit and both proceed, briefly overshooting. For most teams, a small overage is acceptable. If you need strict enforcement, use SELECT FOR UPDATE or maintain the counter in Redis with atomic increment. Second, estimation before the call is inherently imprecise. Budget the worst case, not the average.

Rogue Agent Detection

Agents are the failure mode that catches teams off guard. A well-behaved agent processes a document, makes three LLM calls, and exits. A rogue agent enters a retry loop, hallucinates tool arguments, spins on error correction, and makes 400 LLM calls in ten minutes. At GPT-4o pricing, that is real money.

The detection pattern uses a sliding-window counter per agent. If an agent exceeds a call rate threshold, it gets suspended and an alert fires.

interface AgentCostWindow {
  agentId: string;
  windowStartMs: number;
  windowDurationMs: number;
  callCount: number;
  totalCostUsd: number;
  maxCallsPerWindow: number;
  maxCostPerWindowUsd: number;
}

async function recordAgentCall(
  agentId: string,
  costUsd: number,
  redis: RedisClient
): Promise<{ suspended: boolean; reason?: string }> {
  const now = Date.now();
  const windowKey = `agent:window:${agentId}`;
  const windowDurationMs = 60_000; // 1-minute sliding window

  const pipe = redis.pipeline();
  pipe.hIncrBy(windowKey, "callCount", 1);
  pipe.hIncrByFloat(windowKey, "totalCostUsd", costUsd);
  pipe.hSetNX(windowKey, "windowStartMs", now.toString());
  pipe.expire(windowKey, Math.ceil(windowDurationMs / 1000) * 2);
  await pipe.exec();

  const window = await redis.hGetAll(windowKey);
  const windowAge = now - parseInt(window.windowStartMs ?? "0");

  // Reset window if it has expired
  if (windowAge > windowDurationMs) {
    await redis.del(windowKey);
    return { suspended: false };
  }

  const callCount = parseInt(window.callCount ?? "0");
  const totalCostUsd = parseFloat(window.totalCostUsd ?? "0");

  // Thresholds: more than 50 calls per minute or $5 per minute
  if (callCount > 50) {
    await suspendAgent(agentId, redis);
    return { suspended: true, reason: `call_rate_exceeded: ${callCount} calls in window` };
  }

  if (totalCostUsd > 5.0) {
    await suspendAgent(agentId, redis);
    return { suspended: true, reason: `cost_rate_exceeded: $${totalCostUsd.toFixed(2)} in window` };
  }

  return { suspended: false };
}

async function suspendAgent(agentId: string, redis: RedisClient): Promise<void> {
  await redis.set(`agent:suspended:${agentId}`, "1", { ex: 3600 });
  // Fire alert to on-call channel
  await fireAlert({
    severity: "high",
    title: `Agent suspended: runaway token consumption`,
    body: `Agent ${agentId} exceeded cost or call rate thresholds.`,
    channel: "ai-cost-alerts",
  });
}

async function isAgentSuspended(agentId: string, redis: RedisClient): Promise<boolean> {
  return (await redis.get(`agent:suspended:${agentId}`)) === "1";
}

Add the suspension check as the first gate in your agent dispatch path. A suspended agent gets a clear error that your orchestration layer can surface to the human who needs to investigate.

Shadow AI Spend Detection

Shadow AI is spend that bypasses your central LLM gateway. An engineer adds their personal OpenAI API key to a service. A team signs up for a separate Anthropic account. A feature sends prompts directly to a provider without going through the budget layer.

You cannot detect shadow API keys at the network level easily, but you can detect provider accounts that appear in your infrastructure without being registered in your cost ledger.

The practical approach is to centralize API keys through a secrets manager and audit key issuance. Every provider API key in your infrastructure should trace to an entry in a key registry.

interface ProviderKeyRegistration {
  keyId: string; // e.g. "sk-...abc" last 8 chars as identifier
  provider: string;
  teamId: string;
  featureId: string;
  registeredBy: string;
  registeredAt: Date;
  expiresAt?: Date;
  purpose: string;
}

// CI hook: scan deployed environment variables for unregistered LLM keys
async function auditDeployedKeys(
  deploymentEnvVars: Record<string, string>,
  registry: ProviderKeyRegistration[],
  db: DatabaseClient
): Promise<{ unregistered: string[]; expired: string[] }> {
  const knownKeyIds = new Set(registry.map((r) => r.keyId));
  const unregistered: string[] = [];
  const expired: string[] = [];
  const now = new Date();

  const llmKeyPatterns = [
    /^sk-[A-Za-z0-9]{20,}/, // OpenAI
    /^sk-ant-[A-Za-z0-9-]{20,}/, // Anthropic
    /^AIza[A-Za-z0-9_-]{35}/, // Google
  ];

  for (const [envVar, value] of Object.entries(deploymentEnvVars)) {
    const isLLMKey = llmKeyPatterns.some((p) => p.test(value));
    if (!isLLMKey) continue;

    const keyId = value.slice(-8); // last 8 chars as identifier
    if (!knownKeyIds.has(keyId)) {
      unregistered.push(envVar);
    }

    const registration = registry.find((r) => r.keyId === keyId);
    if (registration?.expiresAt && registration.expiresAt < now) {
      expired.push(envVar);
    }
  }

  return { unregistered, expired };
}

Run this audit in your CI pipeline on every deployment. An unregistered key is a blocker. This does not catch keys that are dynamically injected at runtime by a developer’s local machine, but it catches the most common case: a key committed to config or set in a deployment environment without going through your provisioning process.

Zombie Resource Cleanup

Zombie resources in AI infrastructure are provisioned resources that nobody is using but that accrue cost: fine-tuned model endpoints left running, vector database collections that are never queried, embedding jobs that keep polling, retrieval pipelines serving stale data. At moderate scale these add up to hundreds of dollars per month.

The detection pattern uses access timestamps. Every resource has a lastAccessedAt field. A daily job checks for resources with stale access timestamps and flags them.

interface AIResource {
  id: string;
  type: "fine-tuned-endpoint" | "vector-collection" | "embedding-job" | "batch-job";
  teamId: string;
  provider: string;
  externalId: string; // provider's resource ID
  monthlyCostEstimateUsd: number;
  createdAt: Date;
  lastAccessedAt: Date;
  status: "active" | "zombie-candidate" | "decommissioned";
}

async function runZombieDetection(
  db: DatabaseClient,
  notifier: AlertNotifier
): Promise<void> {
  const staleThresholdDays = 14;
  const staleDate = new Date();
  staleDate.setDate(staleDate.getDate() - staleThresholdDays);

  const candidates = await db.query<AIResource>(
    `SELECT *
     FROM ai_resources
     WHERE status = 'active'
       AND last_accessed_at < $1
       AND monthly_cost_estimate_usd > 0
     ORDER BY monthly_cost_estimate_usd DESC`,
    [staleDate]
  );

  if (candidates.length === 0) return;

  const totalMonthlyWaste = candidates.reduce(
    (sum, r) => sum + r.monthlyCostEstimateUsd,
    0
  );

  // Mark as zombie candidates
  const ids = candidates.map((r) => r.id);
  await db.execute(
    `UPDATE ai_resources SET status = 'zombie-candidate' WHERE id = ANY($1)`,
    [ids]
  );

  await notifier.send({
    channel: "ai-cost-alerts",
    severity: "medium",
    title: `${candidates.length} zombie AI resources detected`,
    body:
      `Estimated monthly waste: $${totalMonthlyWaste.toFixed(2)}. ` +
      `Top offender: ${candidates[0].type} for team ${candidates[0].teamId} ` +
      `($${candidates[0].monthlyCostEstimateUsd.toFixed(2)}/month, ` +
      `last accessed ${Math.floor((Date.now() - candidates[0].lastAccessedAt.getTime()) / 86400000)} days ago).`,
  });
}

Update lastAccessedAt on every real query to the resource. The update can be fire-and-forget: accuracy to within an hour is sufficient for zombie detection.

Multi-Model Billing Consolidation

If you use more than one provider, reconciling the bills is a manual process unless you automate it. Each provider has a different billing format, different granularity, and different update lag. The consolidation goal is a single cost view across OpenAI, Anthropic, Google, and any self-hosted or managed inference providers.

The architecture is straightforward: a daily reconciliation job pulls usage reports from each provider’s API, maps them to your internal feature and team taxonomy using the metadata you tagged at call time, and writes the results to a reconciliation table.

interface ProviderUsageReport {
  provider: string;
  periodStart: Date;
  periodEnd: Date;
  lineItems: Array<{
    model: string;
    promptTokens: number;
    completionTokens: number;
    totalCostUsd: number;
  }>;
}

interface ReconciliationEntry {
  provider: string;
  model: string;
  periodStart: Date;
  periodEnd: Date;
  providerReportedCostUsd: number; // from provider API
  ledgerCostUsd: number;           // from your cost ledger
  varianceUsd: number;             // delta; flag if > threshold
  variancePct: number;
}

async function reconcile(
  provider: string,
  providerReport: ProviderUsageReport,
  db: DatabaseClient
): Promise<ReconciliationEntry[]> {
  const entries: ReconciliationEntry[] = [];

  for (const item of providerReport.lineItems) {
    const ledgerRow = await db.queryOne<{ total_cost_usd: number }>(
      `SELECT COALESCE(SUM(cost_usd), 0) AS total_cost_usd
       FROM cost_ledger
       WHERE provider = $1
         AND model = $2
         AND timestamp >= $3
         AND timestamp < $4`,
      [provider, item.model, providerReport.periodStart, providerReport.periodEnd]
    );

    const ledgerCost = ledgerRow?.total_cost_usd ?? 0;
    const variance = item.totalCostUsd - ledgerCost;
    const variancePct = ledgerCost > 0 ? Math.abs(variance) / ledgerCost : 1;

    entries.push({
      provider,
      model: item.model,
      periodStart: providerReport.periodStart,
      periodEnd: providerReport.periodEnd,
      providerReportedCostUsd: item.totalCostUsd,
      ledgerCostUsd: ledgerCost,
      varianceUsd: variance,
      variancePct,
    });
  }

  // Flag large variances for investigation (>10% or >$50 absolute)
  const flagged = entries.filter(
    (e) => Math.abs(e.varianceUsd) > 50 || e.variancePct > 0.1
  );
  if (flagged.length > 0) {
    console.warn(`Reconciliation variance detected:`, flagged);
  }

  return entries;
}

Variance in reconciliation is usually caused by: requests that bypassed your middleware (shadow spend), provider billing lag (usage from one day appearing in the next day’s report), or pricing table mismatches (your model pricing table is stale). Each cause points to a different fix.

Tradeoffs

Approach	Enforcement Strength	Overhead	Failure Mode
Hard stop at budget limit	Strict: no overage	One DB read per request (~1ms)	Production outage if budget runs dry mid-feature
Soft stop with model degradation	Moderate: cost reduction, not elimination	Same overhead	Teams notice quality drop; less likely to alert
Async alerting only	None: observability only	Minimal	Bill surprise; no automated recovery
Redis atomic counter	Strict with high concurrency	Two Redis calls per request	Redis outage disables enforcement
Pre-period budget allocation	Strict: prevents inter-period drift	Budget must be planned ahead	Inflexible for spiky workloads

Cost Dashboard Structure

A cost dashboard for AI infrastructure needs three views. The daily view shows spend by team and by feature for the current billing period, with yesterday vs. the trailing 7-day average. The agent view shows per-agent cost over the last 24 hours, sorted by descending spend, with call count and average cost per call. The alert view shows active budget threshold breaches, suspended agents, and zombie candidates.

Cost-per-query is the unit economics metric worth tracking separately. Take the total LLM cost for a feature and divide by the number of successful user-facing queries it served. If that number rises without a corresponding rise in quality metrics, you are spending more per user without giving them more. That is the signal that routing logic or context assembly needs review.

Production Checklist

Before shipping your LLM cost governance layer:

• Every LLM call tagged with teamId, featureId, and agentId
• Cost ledger persisting to durable storage (not just metrics)
• Budget enforcement middleware in the call path, not as a sidecar
• Agent suspension check runs before every agent dispatch
• Daily zombie detection job with Slack or PagerDuty output
• Provider API keys inventoried in a registry; CI audit on deploy
• Monthly reconciliation against provider billing API
• Cost-per-query tracked per feature alongside latency and error rate
• Budget alerts firing at 50%, 75%, and 90% utilization, not just at 100%

The 100% alert is too late. By the time a budget is exhausted, the team that owns the feature needs to have already been in a conversation about usage patterns.

Governance for AI infrastructure is not fundamentally different from governance for any other shared resource. The difference is the cost curve: a misconfigured agent can produce a spike in minutes that a misconfigured database connection pool would take days to create. The speed of the feedback loop needs to match the speed of the failure mode.