Building a Real-Time Content Moderation Pipeline: Text Classification, Image Safety, and Human Review Workflows in Production

Most content moderation failures are not model failures. They are pipeline failures. The model predicted correctly; the wrong signal was routed to the wrong stage, or the decision was never reviewed, or the threshold was tuned for demo traffic and never adjusted after launch. Building a moderation system that works under production load means thinking about the pipeline before you think about the models.

This article walks through the architecture of a production content moderation system: the multi-layer approach to text classification, image safety with hash matching and NSFW classifiers, human review queue design, and the feedback loop that keeps accuracy from drifting over time.

The Failure Modes Worth Naming First

Before any architecture decision, name what you are protecting against:

False negatives at scale: A 1% miss rate sounds tolerable until you are processing 10,000 posts per hour. That is 100 harmful pieces of content reaching users every hour.

False positives that choke the platform: Overly aggressive classifiers suppress legitimate content, frustrate users, and generate appeals that overwhelm the review queue.

Latency that makes the pipeline useless: A moderation check that takes 4 seconds is a check that engineers will bypass. Real-time pipelines need a decision in under 300ms for synchronous blocking, or they run asynchronously with a brief hold window.

Model drift without visibility: Your classifier was accurate on launch-day traffic. Six months later, new slang, new contexts, and new user demographics have shifted the distribution. Without a measurement loop, you will not know until the PR disaster.

A well-designed pipeline addresses all four. A fast, cheap pre-filter handles the clear cases. An ML classifier handles the middle tier. An LLM handles nuanced edge cases. Humans handle the irreducible ambiguity. A feedback loop closes the gap between all of them.

Layer 1: Rule-Based Pre-Filters

The fastest moderation is the moderation that never touches a model. A rule-based pre-filter runs in microseconds and handles the unambiguous cases: exact-match blocklists, known CSAM hashes (PhotoDNA / NCMEC hash database), URL reputation feeds, and regex patterns for phone numbers, emails, and payment card data.

interface PreFilterResult {
  action: "block" | "pass" | "flag";
  reason?: string;
  matchedRule?: string;
}

interface PreFilterConfig {
  blocklist: Set<string>;
  hashlist: Set<string>; // PhotoDNA or MD5 hash set
  urlReputationFeed: Map<string, "malicious" | "suspicious">;
  patterns: Array<{ name: string; regex: RegExp; action: "block" | "flag" }>;
}

function runPreFilter(
  content: { text?: string; imageHash?: string; urls?: string[] },
  config: PreFilterConfig
): PreFilterResult {
  // Exact hash match -- always block, no model needed
  if (content.imageHash && config.hashlist.has(content.imageHash)) {
    return { action: "block", reason: "known_hash_match", matchedRule: "hashlist" };
  }

  if (content.text) {
    const normalized = content.text.toLowerCase().trim();

    // Blocklist exact match
    if (config.blocklist.has(normalized)) {
      return { action: "block", reason: "blocklist_match", matchedRule: "blocklist" };
    }

    // Pattern matching
    for (const { name, regex, action } of config.patterns) {
      if (regex.test(content.text)) {
        return { action, reason: `pattern_${name}`, matchedRule: name };
      }
    }
  }

  // URL reputation check
  if (content.urls) {
    for (const url of content.urls) {
      const rep = config.urlReputationFeed.get(url);
      if (rep === "malicious") return { action: "block", reason: "malicious_url", matchedRule: "url_rep" };
      if (rep === "suspicious") return { action: "flag", reason: "suspicious_url", matchedRule: "url_rep" };
    }
  }

  return { action: "pass" };
}

The pre-filter runs before any model inference. If it returns block, the content never touches a GPU. If it returns flag, the content goes to the human queue directly, bypassing ML classification entirely. This is intentional: for known-bad content, you want human confirmation, not ML confidence.

The blocklist and hash list update on a schedule from external feeds (NCMEC for CSAM hashes, commercial threat feeds for URLs). Never hard-code them into your application.

Layer 2: ML Classification

Content that passes the pre-filter goes to ML classification. At this layer you have two choices: use an API or run a custom model. The honest answer is to use an API until you have a volume problem or a distribution problem that the API cannot handle.

OpenAI Moderation API covers hate, harassment, self-harm, sexual content, and violence. It is fast, free at moderate volume, and returns category-level scores you can threshold independently. It does not cover your domain-specific policy violations.

Perspective API (Google Jigsaw) specializes in toxicity, insult, and threat in conversational text. It handles multilingual content better than most alternatives at this tier. It has a rate limit that becomes a constraint around 1 QPS without quota increases.

AWS Rekognition handles image moderation with a hierarchical taxonomy (explicit nudity, violence, visually disturbing content). It returns confidence scores per category, and you set your own threshold. Latency runs 200-400ms depending on image size.

import OpenAI from "openai";

interface ClassificationResult {
  scores: Record<string, number>;
  flagged: boolean;
  provider: string;
  latencyMs: number;
}

const openai = new OpenAI();

async function classifyText(text: string): Promise<ClassificationResult> {
  const start = Date.now();

  const response = await openai.moderations.create({ input: text });
  const result = response.results[0];

  return {
    scores: result.category_scores as Record<string, number>,
    flagged: result.flagged,
    provider: "openai-moderation",
    latencyMs: Date.now() - start,
  };
}

// Perspective API call (via REST, no official SDK)
async function classifyToxicity(text: string, apiKey: string): Promise<ClassificationResult> {
  const start = Date.now();

  const response = await fetch(
    `https://commentanalyzer.googleapis.com/v1alpha1/comments:analyze?key=${apiKey}`,
    {
      method: "POST",
      headers: { "Content-Type": "application/json" },
      body: JSON.stringify({
        comment: { text },
        requestedAttributes: {
          TOXICITY: {},
          SEVERE_TOXICITY: {},
          INSULT: {},
          THREAT: {},
        },
      }),
    }
  );

  const data = await response.json();
  const scores: Record<string, number> = {};

  for (const [attr, val] of Object.entries(data.attributeScores as Record<string, any>)) {
    scores[attr.toLowerCase()] = val.summaryScore.value;
  }

  return {
    scores,
    flagged: scores["toxicity"] > 0.7,
    provider: "perspective",
    latencyMs: Date.now() - start,
  };
}

Run both APIs in parallel where your policy requires it. The results are complementary: OpenAI catches more categories, Perspective catches more subtlety in conversational toxicity.

Where the ML layer breaks down: context. “I’m going to kill this bug” is not a threat. “That’s so gay” may or may not be a slur depending on the speaker and context. A classifier trained on labeled examples cannot reliably resolve these. That is the job of the next layer.

Layer 3: LLM-Based Nuance Detection

For content that the ML classifier scores in the ambiguous range (say, 0.4 to 0.7 on any critical category), you route to an LLM with a structured prompt that includes the content, the relevant policy section, and the surrounding context (username, post history signals, community context).

interface ModerationContext {
  content: string;
  authorReputation: "new" | "established" | "flagged";
  communityContext: string; // e.g. "gaming community", "medical advice forum"
  priorViolations: number;
}

interface LLMDecision {
  action: "allow" | "remove" | "human_review";
  confidence: number; // 0-1
  reasoning: string;
  violatedPolicies: string[];
}

async function llmNuanceCheck(
  ctx: ModerationContext,
  openai: OpenAI
): Promise<LLMDecision> {
  const response = await openai.chat.completions.create({
    model: "gpt-4o-mini", // cheaper than gpt-4o for volume
    response_format: { type: "json_object" },
    messages: [
      {
        role: "system",
        content: `You are a content moderation assistant. Evaluate the following content against platform policy.
Return a JSON object with:
- action: "allow" | "remove" | "human_review"
- confidence: number between 0 and 1
- reasoning: one sentence explaining the decision
- violatedPolicies: array of policy names (empty if action is allow)

Policy categories: hate_speech, harassment, threats, sexual_content, self_harm, spam, misinformation.
Context matters. Consider the community and author history.`,
      },
      {
        role: "user",
        content: `Content: "${ctx.content}"
Author reputation: ${ctx.authorReputation}
Community: ${ctx.communityContext}
Prior violations: ${ctx.priorViolations}`,
      },
    ],
  });

  return JSON.parse(response.choices[0].message.content!) as LLMDecision;
}

Use gpt-4o-mini for this layer unless your policy violations require deep reasoning (CSAM-adjacent edge cases, coordinated inauthentic behavior). The cost difference between mini and full 4o is meaningful at volume: at 1 million nuance checks per day, it is the difference between $150/day and $1,500/day.

Set a hard timeout of 2 seconds on the LLM call. If it times out, route to human review. Never let a timeout become an implicit allow.

Image Safety: Classifiers and Hash Matching

For images, the pipeline has two distinct passes: hash matching (pre-filter layer, covered above) and perceptual classification.

Hash matching catches exact re-uploads of known-bad content. It does not catch new content or edited images. Perceptual classification catches new content but requires a threshold decision.

AWS Rekognition is the default choice for most teams: it handles JPEG, PNG, GIF, and WebP, returns a confidence score per category, and integrates cleanly with S3. The threshold you set matters more than the model choice. At 80% confidence, you will block a meaningful amount of legitimate content (beach photos, medical illustrations, art). At 95%, you will miss borderline content. Most teams run 85% as a blocking threshold and route 70-85% to human review.

import {
  RekognitionClient,
  DetectModerationLabelsCommand,
} from "@aws-sdk/client-rekognition";

interface ImageSafetyResult {
  safe: boolean;
  labels: Array<{ name: string; confidence: number; parentName?: string }>;
  action: "block" | "review" | "allow";
}

const rekognition = new RekognitionClient({ region: "us-east-1" });

const BLOCK_THRESHOLD = 85;
const REVIEW_THRESHOLD = 70;

async function checkImageSafety(
  s3Bucket: string,
  s3Key: string
): Promise<ImageSafetyResult> {
  const command = new DetectModerationLabelsCommand({
    Image: { S3Object: { Bucket: s3Bucket, Name: s3Key } },
    MinConfidence: REVIEW_THRESHOLD,
  });

  const response = await rekognition.send(command);
  const labels = (response.ModerationLabels ?? []).map((l) => ({
    name: l.Name ?? "",
    confidence: l.Confidence ?? 0,
    parentName: l.ParentName,
  }));

  const highConfidenceBlock = labels.some(
    (l) => l.confidence >= BLOCK_THRESHOLD && !l.parentName // top-level category
  );

  const reviewCandidate = labels.some(
    (l) => l.confidence >= REVIEW_THRESHOLD && l.confidence < BLOCK_THRESHOLD
  );

  return {
    safe: labels.length === 0,
    labels,
    action: highConfidenceBlock ? "block" : reviewCandidate ? "review" : "allow",
  };
}

The parentName check matters. Rekognition returns both top-level categories (“Explicit Nudity”) and subcategories (“Graphic Female Nudity”). Blocking on a subcategory without considering the parent can give you inconsistent policy application. Anchor your threshold decisions on the parent categories.

Orchestrating the Full Pipeline

The layers above need a coordinator that routes content through them in the right order, enforces timeouts, and records every decision with enough context to audit and improve.

type ContentType = "text" | "image" | "mixed";

interface ModerationInput {
  contentId: string;
  type: ContentType;
  text?: string;
  imageS3Key?: string;
  imageHash?: string;
  imageS3Bucket?: string;
  authorReputation: "new" | "established" | "flagged";
  communityContext: string;
  priorViolations: number;
  urls?: string[];
}

interface ModerationDecision {
  contentId: string;
  action: "allow" | "block" | "human_review";
  stage: "pre_filter" | "ml_classifier" | "llm_nuance" | "image_safety";
  confidence?: number;
  reasoning?: string;
  durationMs: number;
}

async function moderateContent(
  input: ModerationInput,
  config: PreFilterConfig,
  openai: OpenAI,
  perspectiveApiKey: string
): Promise<ModerationDecision> {
  const start = Date.now();

  // Stage 1: Pre-filter (always runs)
  const preFilterResult = runPreFilter(
    { text: input.text, imageHash: input.imageHash, urls: input.urls },
    config
  );

  if (preFilterResult.action === "block") {
    return { contentId: input.contentId, action: "block", stage: "pre_filter", durationMs: Date.now() - start };
  }

  if (preFilterResult.action === "flag") {
    return { contentId: input.contentId, action: "human_review", stage: "pre_filter", durationMs: Date.now() - start };
  }

  // Stage 2: ML classification and image safety (parallel where possible)
  const tasks: Promise<unknown>[] = [];

  let textScore: ClassificationResult | null = null;
  let imageResult: ImageSafetyResult | null = null;

  if (input.text) {
    tasks.push(
      classifyText(input.text).then((r) => { textScore = r; })
    );
  }

  if (input.imageS3Key && input.imageS3Bucket) {
    tasks.push(
      checkImageSafety(input.imageS3Bucket, input.imageS3Key).then((r) => { imageResult = r; })
    );
  }

  await Promise.allSettled(tasks);

  if (imageResult?.action === "block") {
    return { contentId: input.contentId, action: "block", stage: "image_safety", durationMs: Date.now() - start };
  }

  if (imageResult?.action === "review") {
    return { contentId: input.contentId, action: "human_review", stage: "image_safety", durationMs: Date.now() - start };
  }

  if (textScore?.flagged) {
    // Check if high-confidence block or ambiguous (route to LLM)
    const maxScore = Math.max(...Object.values(textScore.scores));

    if (maxScore > 0.85) {
      return { contentId: input.contentId, action: "block", stage: "ml_classifier", confidence: maxScore, durationMs: Date.now() - start };
    }

    // Stage 3: LLM nuance check for ambiguous range
    if (maxScore > 0.4 && input.text) {
      const llmDecision = await llmNuanceCheck(
        {
          content: input.text,
          authorReputation: input.authorReputation,
          communityContext: input.communityContext,
          priorViolations: input.priorViolations,
        },
        openai
      );

      return {
        contentId: input.contentId,
        action: llmDecision.action === "allow" ? "allow" : llmDecision.action === "remove" ? "block" : "human_review",
        stage: "llm_nuance",
        confidence: llmDecision.confidence,
        reasoning: llmDecision.reasoning,
        durationMs: Date.now() - start,
      };
    }
  }

  return { contentId: input.contentId, action: "allow", stage: "ml_classifier", durationMs: Date.now() - start };
}

Every decision gets a stage field. This is how you diagnose accuracy problems later: if you see high false positive rates and they are all from stage: llm_nuance, you know where to tune.

Human Review Queue Design

Human review is not the failure state of an automated pipeline. It is a deliberate stage for the content that genuinely requires human judgment. Design it accordingly.

The queue needs priority tiers. Content that was tentatively published pending review (allowed with a hold window) is higher priority than content that was pre-emptively hidden. Content from users with prior violations is higher priority than content from established accounts. Time-sensitive content (live events, trending topics) is higher priority than archive content.

Every item in the queue needs the full decision context: what triggered the review, which stage flagged it, the raw model scores, and the policy sections that may apply. Reviewers should never be asked to make a decision blind.

Track these metrics per reviewer: decisions per hour, agreement rate with automated decisions, override rate, and inter-rater agreement on a shared calibration set. If two reviewers are making opposite decisions on the same content type, you have a policy clarity problem, not a reviewer performance problem.

The queue also doubles as your training data pipeline. Every human decision is a labeled example. Feed confirmed human decisions back into your ML classifier retraining cycle. Set a minimum batch size (500 new labeled examples) before triggering a retrain, and always evaluate on a held-out set before promoting a new model version.

The Feedback Loop

The feedback loop is what separates a moderation system that works at launch from one that keeps working six months later.

The loop has four steps: collect decisions (both automated and human), measure accuracy against the human-reviewed subset, identify failure modes (which categories, which content types, which communities), and retrain or re-threshold.

Re-thresholding is faster than retraining. If you notice that your false negative rate on harassment is climbing but the model scores look reasonable, the problem is often that the threshold that worked at launch is wrong for the current traffic distribution. Shift the threshold on that category before committing to a full retrain.

For full retraining, use the human-reviewed decisions from the past 90 days weighted toward recent data. Set an automated evaluation gate: if the new model version does not improve F1 on your policy-critical categories by at least 2%, do not promote it. A model that is marginally worse on one category but better on another is not an obvious win.

Build vs. API: Honest Tradeoffs

Dimension	OpenAI Moderation	Perspective API	AWS Rekognition	Custom Model
Setup time	Minutes	Hours	Hours	Weeks to months
Text coverage	Broad (hate, violence, sexual, self-harm)	Toxicity-focused, multilingual	Not designed for text	Whatever you train
Image coverage	None	None	Strong (NSFW, violence)	Whatever you train
Domain-specific policy	No	No	No	Yes
Cost at 1M req/day	Free (text moderation)	~$1/1K after quota	~$1/1K images	Infra + engineering time
Latency	100-200ms	200-400ms	200-400ms	Depends on hardware
Data privacy	Content sent to OpenAI	Content sent to Google	Content sent to AWS	You control
Accuracy on your distribution	Calibrated on generic data	Calibrated on generic data	Calibrated on generic data	Calibrated on your data
Multilingual	Limited	Strong	Limited	Depends on training data

The practical recommendation: start with OpenAI Moderation for text and AWS Rekognition for images. Add Perspective API if you have a conversational product with multilingual users. Build a custom model only after you have accumulated enough labeled data from your human review queue and have identified specific gaps the APIs cannot fill.

Custom models win on distribution fit and policy specificity. They lose on maintenance burden and the cold-start problem: you need enough labeled data from your platform’s specific traffic distribution to train a model that outperforms a well-tuned API.

Production Considerations

Idempotency: Every content item needs a stable contentId. If the pipeline runs twice on the same item (retry after failure, re-upload detection), the second run should produce the same decision or route to human review rather than double-blocking.

Asynchronous vs. synchronous: Synchronous blocking (decision before content appears) gives the best user experience for writes but requires the pipeline to complete in under 300ms. Asynchronous with a hold window (content appears briefly, removed if flagged) works for high-volume feeds where latency is non-negotiable. Never publish content and then fail to review it; the hold window must be enforced.

Observability: Track false positive rate (user appeals that result in reinstatement), false negative rate (user reports on content that passed), stage distribution (what fraction of content reaches each layer), and queue depth and time-to-review. Alert on queue depth growth: a growing queue means either volume spiked or reviewer capacity dropped, both of which need immediate response.

Regulatory considerations: In the EU, the Digital Services Act imposes specific requirements on large platforms: appeals processes, transparency reports, and human review for automated decisions. Build the appeals mechanism and the audit log before you need them, not after.

Rate limits at peak traffic: Every external API has rate limits. Pre-calculate your burst capacity and have a graceful degradation strategy: if the LLM layer is rate-limited, route those items directly to human review rather than allowing them through.

Closing

Content moderation systems fail at the architecture level before they fail at the model level. The pre-filter keeps the cheap path cheap. The ML classifier handles volume. The LLM handles nuance. Humans handle the irreducible edge cases. The feedback loop keeps all of it calibrated. Each stage has to know exactly what to do when the next stage is unavailable.

Start with the APIs, build the human review queue from day one, instrument every stage decision, and treat accuracy as an ongoing operational concern rather than a launch-day metric.