Building an AI-Powered Code Review Pipeline: Static Analysis, LLM Review, and Automated Feedback

Most LLM code review demos show a prompt and a response. They skip the part where your pipeline ingests a 4,000-line diff, picks which files matter, stays inside a token budget, avoids posting hallucinated suggestions, and doesn’t annoy engineers with noise on every pull request.

This article covers the full pipeline: webhook ingestion, diff parsing, static analysis as a pre-filter, LLM review stage, comment posting, prompt design, token management, and calibrating severity. Each section includes TypeScript. The goal is a pipeline you can actually run in CI.

---

Pipeline Architecture

The pipeline has five sequential stages:

1. Webhook ingestion: receive and validate GitHub PR events
2. Diff parsing: extract changed hunks per file
3. Static analysis: lint, type-check, security scan; filter out obvious issues before the LLM sees them
4. LLM review: send relevant diff context with structured prompts; receive structured feedback
5. Comment posting: post inline comments, suppress low-signal ones, respect existing review threads

The static analysis stage is not optional. Running the LLM on a diff that already has 40 ESLint violations wastes tokens and produces a review that reads like a linter output. Static analysis clears the low-hanging fruit. The LLM then focuses on logic errors, race conditions, security implications, and architectural concerns that linters cannot catch.

---

Stage 1: Webhook Ingestion

import { createHmac, timingSafeEqual } from "crypto";
import type { IncomingMessage, ServerResponse } from "http";

interface GitHubPullRequestEvent {
  action: "opened" | "synchronize" | "reopened";
  pull_request: {
    number: number;
    head: { sha: string; ref: string };
    base: { sha: string; ref: string };
    additions: number;
    deletions: number;
    changed_files: number;
  };
  repository: {
    full_name: string;
    clone_url: string;
  };
  installation?: { id: number };
}

function verifyWebhookSignature(
  payload: Buffer,
  signature: string,
  secret: string
): boolean {
  const expected = `sha256=${createHmac("sha256", secret)
    .update(payload)
    .digest("hex")}`;
  const sig = Buffer.from(signature);
  const exp = Buffer.from(expected);
  if (sig.length !== exp.length) return false;
  return timingSafeEqual(sig, exp);
}

export async function handleWebhook(
  req: IncomingMessage,
  res: ServerResponse
): Promise<void> {
  const chunks: Buffer[] = [];
  for await (const chunk of req) chunks.push(chunk as Buffer);
  const body = Buffer.concat(chunks);

  const signature = req.headers["x-hub-signature-256"] as string;
  if (!verifyWebhookSignature(body, signature, process.env.GITHUB_WEBHOOK_SECRET!)) {
    res.writeHead(401).end("invalid signature");
    return;
  }

  const event = JSON.parse(body.toString()) as GitHubPullRequestEvent;
  if (!["opened", "synchronize", "reopened"].includes(event.action)) {
    res.writeHead(200).end("ignored");
    return;
  }

  // Guard against very large PRs before queuing
  if (event.pull_request.changed_files > 50) {
    await postPrComment(event, "PR too large for automated review (>50 files). Review manually.");
    res.writeHead(200).end("skipped: too large");
    return;
  }

  await enqueueReview(event);
  res.writeHead(202).end("queued");
}

The size guard matters. PRs over 50 changed files rarely benefit from automated LLM review: the context is too fragmented, and the signal-to-noise ratio drops sharply. Post a comment explaining the skip rather than silently doing nothing.

---

Stage 2: Diff Parsing

GitHub’s REST API returns diffs in unified diff format. You need to extract changed hunks per file, not the entire diff as a blob.

interface DiffHunk {
  header: string;        // e.g. "@@ -10,7 +10,9 @@"
  oldStart: number;
  newStart: number;
  lines: string[];       // lines prefixed with +, -, or space
}

interface FileDiff {
  path: string;
  status: "added" | "modified" | "deleted" | "renamed";
  additions: number;
  deletions: number;
  hunks: DiffHunk[];
  language: string;
}

function parseUnifiedDiff(rawDiff: string): FileDiff[] {
  const files: FileDiff[] = [];
  const fileBlocks = rawDiff.split(/^diff --git /m).filter(Boolean);

  for (const block of fileBlocks) {
    const lines = block.split("\n");
    const pathMatch = lines[0].match(/a\/(.*) b\/(.*)/);
    if (!pathMatch) continue;

    const path = pathMatch[2];
    const language = inferLanguage(path);
    const hunks: DiffHunk[] = [];
    let currentHunk: DiffHunk | null = null;
    let additions = 0;
    let deletions = 0;

    for (const line of lines) {
      const hunkHeader = line.match(/^@@ -(\d+)(?:,\d+)? \+(\d+)(?:,\d+)? @@/);
      if (hunkHeader) {
        if (currentHunk) hunks.push(currentHunk);
        currentHunk = {
          header: line,
          oldStart: parseInt(hunkHeader[1], 10),
          newStart: parseInt(hunkHeader[2], 10),
          lines: [],
        };
        continue;
      }
      if (currentHunk) {
        currentHunk.lines.push(line);
        if (line.startsWith("+")) additions++;
        if (line.startsWith("-")) deletions++;
      }
    }
    if (currentHunk) hunks.push(currentHunk);

    const status = lines.some((l) => l.startsWith("new file"))
      ? "added"
      : lines.some((l) => l.startsWith("deleted file"))
      ? "deleted"
      : "modified";

    files.push({ path, status, additions, deletions, hunks, language });
  }

  return files;
}

function inferLanguage(path: string): string {
  const ext = path.split(".").pop() ?? "";
  const map: Record<string, string> = {
    ts: "typescript", tsx: "typescript", js: "javascript",
    jsx: "javascript", py: "python", go: "go", rs: "rust",
    sql: "sql", yaml: "yaml", yml: "yaml",
  };
  return map[ext] ?? "text";
}

Keep only the changed lines plus 3-5 lines of context on each side per hunk. Sending full file contents is tempting but expensive, and most models handle localized context better than 500-line files anyway.

---

Stage 3: Static Analysis Pre-Filter

Run your existing linters and type-checkers first. Collect their output and attach it to the diff context so the LLM knows what has already been flagged.

import { execFile } from "child_process";
import { promisify } from "util";

const exec = promisify(execFile);

interface StaticIssue {
  file: string;
  line: number;
  column: number;
  severity: "error" | "warning" | "info";
  rule: string;
  message: string;
  source: "eslint" | "tsc" | "semgrep";
}

async function runStaticAnalysis(files: string[]): Promise<StaticIssue[]> {
  const issues: StaticIssue[] = [];

  // ESLint: only on changed files
  try {
    const { stdout } = await exec("npx", [
      "eslint", "--format", "json", "--no-eslintrc",
      "--config", ".eslintrc.ci.json",
      ...files.filter((f) => /\.(ts|tsx|js|jsx)$/.test(f)),
    ]);
    const results = JSON.parse(stdout) as Array<{
      filePath: string;
      messages: Array<{ line: number; column: number; severity: number; ruleId: string; message: string }>;
    }>;
    for (const result of results) {
      for (const msg of result.messages) {
        issues.push({
          file: result.filePath,
          line: msg.line,
          column: msg.column,
          severity: msg.severity === 2 ? "error" : "warning",
          rule: msg.ruleId ?? "unknown",
          message: msg.message,
          source: "eslint",
        });
      }
    }
  } catch (_) { /* eslint exits non-zero on violations */ }

  // TypeScript: type errors only, no emit
  try {
    const { stdout } = await exec("npx", ["tsc", "--noEmit", "--pretty", "false"]);
    const tscLines = stdout.split("\n").filter(Boolean);
    for (const line of tscLines) {
      const match = line.match(/^(.+)\((\d+),(\d+)\): error (TS\d+): (.+)$/);
      if (match) {
        issues.push({
          file: match[1],
          line: parseInt(match[2], 10),
          column: parseInt(match[3], 10),
          severity: "error",
          rule: match[4],
          message: match[5],
          source: "tsc",
        });
      }
    }
  } catch (_) {}

  return issues;
}

The key decision here: if static analysis returns errors, do you still run the LLM review? The answer depends on your team’s workflow. If the CI gate already blocks on lint errors, skip the LLM when there are errors to avoid redundant noise. If lint is advisory, pass the static issues as context so the LLM doesn’t re-flag them.

---

Stage 4: LLM Review

Token Budget Management

A GPT-4o or Claude 3.5 Sonnet call costs roughly $0.003-0.015 per 1,000 output tokens. On a large diff, you can easily burn $0.50 per review if you don’t manage context size. At 50 PRs per day, that’s $25/day or $750/month for a single service.

Set a hard token budget. Prioritize files by surface area (added lines) and file type (skip generated files, lock files, and config-only changes).

function selectFilesForReview(
  files: FileDiff[],
  tokenBudget: number
): FileDiff[] {
  const SKIP_PATTERNS = [
    /\.lock$/, /package-lock\.json$/, /yarn\.lock$/,
    /\.min\.(js|css)$/, /dist\//, /\.generated\./, /node_modules\//,
    /migration\.\d+\.sql$/, // auto-generated migrations
  ];

  const reviewable = files.filter(
    (f) =>
      f.status !== "deleted" &&
      !SKIP_PATTERNS.some((p) => p.test(f.path)) &&
      f.language !== "text"
  );

  // Rough token estimate: 1 token per 3 characters
  const estimateTokens = (f: FileDiff) =>
    Math.ceil(
      f.hunks.flatMap((h) => h.lines).join("\n").length / 3
    );

  // Sort by additions descending (highest signal first)
  reviewable.sort((a, b) => b.additions - a.additions);

  const selected: FileDiff[] = [];
  let used = 0;
  for (const file of reviewable) {
    const cost = estimateTokens(file);
    if (used + cost > tokenBudget) break;
    selected.push(file);
    used += cost;
  }

  return selected;
}

Prompt Design

The prompt structure determines review quality more than model choice. Include: the PR title, a list of static issues already found (so the LLM skips them), and the diff hunks with file paths and line numbers.

interface ReviewComment {
  file: string;
  line: number;
  severity: "critical" | "major" | "minor" | "nit";
  category: "security" | "correctness" | "performance" | "style" | "design";
  message: string;
  suggestion?: string;
}

interface ReviewResponse {
  summary: string;
  comments: ReviewComment[];
}

function buildReviewPrompt(
  prTitle: string,
  files: FileDiff[],
  staticIssues: StaticIssue[]
): string {
  const staticSummary =
    staticIssues.length > 0
      ? `The following issues were already flagged by static analysis. Do not repeat them:\n${staticIssues
          .map((i) => `- ${i.file}:${i.line} [${i.source}/${i.rule}] ${i.message}`)
          .join("\n")}`
      : "No static analysis issues were found.";

  const diffContent = files
    .map((f) => {
      const hunks = f.hunks
        .map((h) => `${h.header}\n${h.lines.join("\n")}`)
        .join("\n");
      return `### File: ${f.path} (${f.language})\n\`\`\`diff\n${hunks}\n\`\`\``;
    })
    .join("\n\n");

  return `You are reviewing a pull request for a production TypeScript codebase.

PR title: ${prTitle}

${staticSummary}

Review the following diff and identify issues that static analysis cannot catch:
- Logic errors, off-by-one errors, incorrect assumptions
- Race conditions, concurrency bugs
- Security vulnerabilities (injection, credential exposure, improper auth checks)
- Missing error handling for operations that can fail
- Incorrect or missing input validation at trust boundaries
- Performance issues with measurable impact (N+1 queries, unbounded loops over external data)
- Architectural concerns that affect maintainability at scale

Do not flag style preferences, minor naming choices, or issues already listed above.
Be specific. Cite line numbers. Suggest a concrete fix when one is clear.

${diffContent}

Respond with valid JSON matching this schema:
{
  "summary": "one or two sentence summary of the overall review",
  "comments": [
    {
      "file": "path/to/file.ts",
      "line": 42,
      "severity": "critical|major|minor|nit",
      "category": "security|correctness|performance|style|design",
      "message": "what the issue is and why it matters",
      "suggestion": "optional: concrete code or approach to fix it"
    }
  ]
}`;
}

async function runLLMReview(
  prompt: string,
  model = "gpt-4o"
): Promise<ReviewResponse> {
  const response = await fetch("https://api.openai.com/v1/chat/completions", {
    method: "POST",
    headers: {
      "Content-Type": "application/json",
      Authorization: `Bearer ${process.env.OPENAI_API_KEY}`,
    },
    body: JSON.stringify({
      model,
      messages: [{ role: "user", content: prompt }],
      temperature: 0.1,       // low temperature: consistent, less creative
      response_format: { type: "json_object" },
      max_tokens: 2048,
    }),
  });

  const data = await response.json() as {
    choices: Array<{ message: { content: string } }>;
  };

  try {
    return JSON.parse(data.choices[0].message.content) as ReviewResponse;
  } catch {
    // Malformed JSON from the model: return empty review rather than crashing
    return { summary: "Review parsing failed.", comments: [] };
  }
}

Temperature 0.1 is intentional. High temperature produces creative suggestions that are plausible-sounding but wrong. For code review you want consistent, conservative output.

---

Stage 5: Comment Posting and Suppression

Posting every comment the LLM generates is how you burn engineer trust in three days. Apply suppression rules before posting.

interface PostingConfig {
  minSeverity: "critical" | "major" | "minor" | "nit";
  maxCommentsPerPr: number;
  suppressCategories: string[];
}

const DEFAULT_CONFIG: PostingConfig = {
  minSeverity: "minor",
  maxCommentsPerPr: 10,
  suppressCategories: ["style"],  // style is covered by formatter
};

function severityRank(s: ReviewComment["severity"]): number {
  return { critical: 4, major: 3, minor: 2, nit: 1 }[s];
}

function filterComments(
  comments: ReviewComment[],
  config: PostingConfig
): ReviewComment[] {
  const minRank = severityRank(config.minSeverity);

  return comments
    .filter((c) => !config.suppressCategories.includes(c.category))
    .filter((c) => severityRank(c.severity) >= minRank)
    .sort((a, b) => severityRank(b.severity) - severityRank(a.severity))
    .slice(0, config.maxCommentsPerPr);
}

async function postReviewComments(
  installationId: number,
  repo: string,
  prNumber: number,
  headSha: string,
  review: ReviewResponse,
  config: PostingConfig = DEFAULT_CONFIG
): Promise<void> {
  const token = await getInstallationToken(installationId);
  const filtered = filterComments(review.comments, config);

  if (filtered.length === 0) {
    // Nothing worth posting: acknowledge with a passing review
    await submitReview(token, repo, prNumber, headSha, {
      body: "Automated review: no issues found.",
      event: "COMMENT",
      comments: [],
    });
    return;
  }

  const reviewComments = filtered.map((c) => ({
    path: c.file,
    line: c.line,
    side: "RIGHT" as const,
    body: formatComment(c),
  }));

  await submitReview(token, repo, prNumber, headSha, {
    body: `Automated review: ${review.summary}\n\n_${filtered.length} issue(s) found. Suppressed ${review.comments.length - filtered.length} lower-priority comment(s)._`,
    event: filtered.some((c) => c.severity === "critical") ? "REQUEST_CHANGES" : "COMMENT",
    comments: reviewComments,
  });
}

function formatComment(c: ReviewComment): string {
  const badge = { critical: "CRITICAL", major: "MAJOR", minor: "MINOR", nit: "NIT" }[c.severity];
  const lines = [`**[${badge}/${c.category}]** ${c.message}`];
  if (c.suggestion) {
    lines.push("", "**Suggestion:**", "```typescript", c.suggestion, "```");
  }
  lines.push("", "_Automated review. Verify before acting on this suggestion._");
  return lines.join("\n");
}

The “verify before acting” footer is not decoration. LLMs hallucinate fixes, especially for unfamiliar APIs, internal helper functions not visible in the diff, and anything involving concurrent state. The disclaimer sets the right expectation without disabling the review entirely.

---

Integrating with CI/CD

The pipeline runs as a GitHub App, not a personal access token. App tokens are scoped to the installation, can be rotated automatically, and support fine-grained repository permissions.

The webhook handler should respond within 10 seconds (GitHub’s timeout) and do actual work asynchronously. Queue the event to a background worker:

// Minimal queue abstraction that works with any backend
interface ReviewQueue {
  enqueue(event: GitHubPullRequestEvent): Promise<void>;
}

// Example with BullMQ or a simple HTTP queue like Quirrel
async function enqueueReview(event: GitHubPullRequestEvent): Promise<void> {
  await reviewQueue.enqueue(event);
}

// Worker: runs the full pipeline
async function processReviewJob(event: GitHubPullRequestEvent): Promise<void> {
  const diff = await fetchPullRequestDiff(event);
  const files = parseUnifiedDiff(diff);
  const selectedFiles = selectFilesForReview(files, 8000); // 8k token budget for diff
  const changedPaths = selectedFiles.map((f) => f.path);

  const staticIssues = await runStaticAnalysis(changedPaths);
  const prompt = buildReviewPrompt(
    event.pull_request.head.ref,
    selectedFiles,
    staticIssues
  );

  const review = await runLLMReview(prompt);
  await postReviewComments(
    event.installation?.id ?? 0,
    event.repository.full_name,
    event.pull_request.number,
    event.pull_request.head.sha,
    review
  );
}

The total latency budget for a useful review is under 60 seconds. Static analysis typically takes 5-20 seconds depending on the codebase. LLM inference adds 5-15 seconds. Post-processing and GitHub API calls add 2-5 seconds. This is achievable within a single CI job.

---

Tradeoffs

Dimension	Approach	Tradeoff
Token budget	Hard ceiling per PR	May skip high-signal files in large PRs. Mitigate by sorting files by addition count first.
Temperature	0.1	Consistent but conservative. May miss creative architectural observations. Acceptable for a safety-focused tool.
Static analysis first	Pre-filter before LLM	Adds 5-20s latency. Reduces token usage 30-60% on lint-heavy diffs. Net positive.
Comment suppression	Min severity + category filter	Risk of suppressing a real issue. Keep logs of suppressed comments for audit.
REQUEST_CHANGES event	Only on critical severity	Blocks merges. Reserve for issues that genuinely require a fix, not every comment.
Model choice	GPT-4o vs Claude 3.5 Sonnet	Similar quality. Claude tends toward more conservative suggestions. GPT-4o is slightly faster on structured output. Test both on your codebase.
Hallucinated suggestions	Footer disclaimer	Engineers dismiss over time. Longer-term: evaluate suggestions against a test suite before posting.
Cost per review	$0.05-0.40 depending on diff size	At 50 PRs/day: $75-600/month. Apply the file filter aggressively.

---

Production Considerations

Idempotency. GitHub delivers webhooks at least once. Store a (pr_number, head_sha) pair in a review log table and skip if already processed. Without this, you post duplicate comments on every redelivery.

Handling hallucinated line numbers. The LLM occasionally generates comments for line numbers that do not exist in the diff. GitHub’s review API rejects these with a 422. Validate that each comment’s line falls within the actual changed range of the referenced file before posting.

Observability. Log: token usage per review (input + output), latency per stage, number of comments generated vs. suppressed, and static issues found. These metrics tell you whether the pipeline is improving or degrading over time. A rising suppression rate with stable generation count suggests prompt drift; a rising hallucination rate on line numbers suggests the model is being given too-sparse context.

Versioning the prompt. Treat your prompt as code. Store it with a version identifier and log which prompt version generated each review. When you update the prompt, you need to compare outputs on historical PRs to verify you improved quality, not just changed it.

Rate limits. GitHub’s API allows 15 requests per minute per installation for review submissions. If you run high-volume repositories, batch comments into a single review submission rather than individual POST calls.

When to suppress entirely. The pipeline should post nothing when: the PR is a merge from a release branch, the PR is opened by a bot (Dependabot, Renovate), the PR title contains [skip review], or the PR has already been reviewed by a human and is awaiting approval. These cases are easy to add to the webhook handler as early returns.

---

Automated code review adds real value when it catches issues humans routinely miss under time pressure: missing error handling, incorrect null checks at trust boundaries, and security patterns that require pattern recognition rather than judgment. It earns trust by being selective, not exhaustive. Post fewer, higher-confidence comments. Leave architectural judgment to humans. The pipeline that posts 3 accurate comments per PR will outlast the one that posts 15 and gets ignored.