Designing a Content Moderation System: Automated Filtering, ML Classification, and Human Review Queues at Scale

Every platform that accepts user-generated content eventually faces the same engineering problem: you need to filter harmful content reliably, at scale, without destroying the user experience with false positives or the reviewer team with burn-out. The surface area is deceptively wide. Text can be toxic, spam, or illegal. Images can be NSFW or CSAM. Videos combine both dimensions. And the regulatory environment is tightening: the EU Digital Services Act (DSA) mandates specific transparency, appeal, and redress obligations. COPPA introduces additional requirements for platforms accessible to minors.

This article walks through the full architecture of a production content moderation pipeline: pre-publication filters, ML classification with confidence-based routing, human review queues, reviewer tooling, appeal workflows, and the feedback loops that keep models calibrated over time.

The Core Problem

A naive implementation blocks on every content submission, runs a classifier, and either publishes or rejects. This breaks down immediately in production:

• Volume: large platforms ingest millions of posts per day; synchronous blocking classification adds latency users notice
• False positive cost: blocking legitimate content erodes trust faster than allowing marginal borderline content
• Model drift: a classifier trained six months ago misses new slang, new meme formats, and adversarial obfuscation techniques
• Legal exposure: under DSA Article 17, platforms must provide a statement of reasons for each content removal and an accessible internal complaints mechanism

The architecture has to be layered. Cheap, fast filters run first. ML models run asynchronously in most cases. Human reviewers handle the uncertain middle. Each layer feeds back into the next.

Layer 1: Pre-Publication Filters

The fastest filters are deterministic and run synchronously before content is persisted. They fall into two categories.

Hash matching uses perceptual hashing to identify known-bad media. PhotoDNA (for CSAM), Microsoft’s Video Indexer, and open-source tools like blockhash generate hashes that tolerate minor transformations (cropping, recompression). You maintain a local hash database seeded from the National Center for Missing and Exploited Children (NCMEC) PhotoDNA cloud service and industry-shared databases like the Hash Sharing Platform.

Keyword blocklists catch explicit slurs, known spam phrases, and regex patterns for things like phone numbers in the wrong context or known phishing domains. These are blunt instruments, but they are fast and have near-zero false negative rate for the patterns they cover.

interface PrePublicationFilter {
  check(content: RawContent): FilterResult;
}

interface FilterResult {
  action: "block" | "flag" | "pass";
  reason?: string;
  matchedRule?: string;
}

interface RawContent {
  type: "text" | "image" | "video";
  text?: string;
  mediaHash?: string;
  authorId: string;
  platformContext: "post" | "comment" | "dm" | "profile";
}

class HashMatchFilter implements PrePublicationFilter {
  constructor(
    private readonly hashDb: HashDatabase,
  ) {}

  check(content: RawContent): FilterResult {
    if (!content.mediaHash) return { action: "pass" };

    const match = this.hashDb.lookup(content.mediaHash);
    if (match?.category === "csam") {
      return { action: "block", reason: "hash_match_csam", matchedRule: match.id };
    }
    if (match?.category === "known_spam_image") {
      return { action: "flag", reason: "hash_match_spam", matchedRule: match.id };
    }
    return { action: "pass" };
  }
}

class KeywordFilter implements PrePublicationFilter {
  private readonly blocklist: RegExp[];
  private readonly flaglist: RegExp[];

  constructor(rules: BlocklistRules) {
    this.blocklist = rules.block.map((r) => new RegExp(r, "i"));
    this.flaglist = rules.flag.map((r) => new RegExp(r, "i"));
  }

  check(content: RawContent): FilterResult {
    if (!content.text) return { action: "pass" };

    for (const pattern of this.blocklist) {
      if (pattern.test(content.text)) {
        return { action: "block", reason: "keyword_blocklist", matchedRule: pattern.source };
      }
    }
    for (const pattern of this.flaglist) {
      if (pattern.test(content.text)) {
        return { action: "flag", reason: "keyword_flaglist", matchedRule: pattern.source };
      }
    }
    return { action: "pass" };
  }
}

Content that hits a hard block never reaches persistence. Content that hits a soft flag is persisted but immediately enqueued for review. Content that passes goes live, with async ML classification running in the background.

Layer 2: ML Classification

ML models run asynchronously against all content, including content that already passed pre-publication filters. The three main classifiers are:

• Text toxicity: detects hate speech, harassment, threats, and self-harm promotion. Common production choices: Perspective API (Google), Jigsaw’s Toxic Comment model, or a fine-tuned DistilBERT/RoBERTa for domain-specific slang.
• NSFW image/video classification: detects adult content and graphic violence. Open-source options include NudeNet, NSFW JS, or fine-tuned CLIP variants. Commercial APIs include Amazon Rekognition Content Moderation and Google Cloud Vision Safe Search.
• Spam and inauthentic behavior: graph-based signals (posting velocity, account age, duplicate content fingerprints) combined with text classifiers for promotional language.

The key design decision is how you use the confidence score. Every classifier produces a probability, not a binary answer. You need three thresholds, not one:

interface ClassificationResult {
  label: "toxic" | "nsfw" | "spam" | "clean";
  confidence: number; // 0.0 - 1.0
  modelVersion: string;
  latencyMs: number;
}

interface RoutingThresholds {
  autoBlock: number;   // e.g., 0.95 — take automated action
  humanReview: number; // e.g., 0.60 — route to human queue
  // below humanReview threshold: leave live or archive quietly
}

function routeByConfidence(
  result: ClassificationResult,
  thresholds: RoutingThresholds,
  contentId: string,
): ModerationAction {
  if (result.confidence >= thresholds.autoBlock) {
    return {
      action: "remove",
      automated: true,
      contentId,
      reason: result.label,
      modelVersion: result.modelVersion,
      confidence: result.confidence,
    };
  }

  if (result.confidence >= thresholds.humanReview) {
    return {
      action: "queue_for_review",
      automated: false,
      contentId,
      reason: result.label,
      modelVersion: result.modelVersion,
      confidence: result.confidence,
    };
  }

  return {
    action: "pass",
    automated: true,
    contentId,
    reason: "below_threshold",
    modelVersion: result.modelVersion,
    confidence: result.confidence,
  };
}

Setting thresholds requires empirical calibration against your platform’s actual false positive and false negative costs. A general-purpose social network weights false positives (wrongly removed content) as high-cost because of user trust. A children’s platform weights false negatives as high-cost because of COPPA and legal exposure. There is no universal threshold. Run precision-recall analysis on a labeled hold-out set from your actual user base before going live.

Layer 3: Human Review Queues

Content that lands in the middle confidence band, or that was pre-flagged, enters a priority-weighted review queue. The queue design matters as much as the ML model.

Priority signals that should elevate items in the queue:

• Content from new accounts (lower trust score)
• Content that received user reports
• Content with higher ML confidence (closer to the auto-block threshold)
• Content in high-reach contexts (viral content, trending hashtags)
• Content categories with legal obligations (potential CSAM always gets immediate escalation)

interface ReviewQueueItem {
  contentId: string;
  contentSnapshot: ContentSnapshot; // immutable copy at time of flag
  mlResults: ClassificationResult[];
  userReports: UserReport[];
  authorTrustScore: number;
  reachScore: number; // estimated impressions if left live
  enqueuedAt: Date;
  priority: number; // derived score; higher = review sooner
  assignedReviewerId?: string;
  slaDeadlineAt: Date;
}

function computePriority(item: Omit<ReviewQueueItem, "priority">): number {
  let score = 0;

  // Higher ML confidence in harmful direction = higher priority
  const maxConfidence = Math.max(...item.mlResults.map((r) => r.confidence));
  score += maxConfidence * 40;

  // More user reports = higher priority
  score += Math.min(item.userReports.length * 5, 20);

  // High reach content should be resolved faster
  score += Math.min(item.reachScore / 1000, 20);

  // New accounts get elevated scrutiny
  if (item.authorTrustScore < 0.3) score += 10;

  // Age decay: older items drift up to prevent queue starvation
  const ageHours = (Date.now() - item.enqueuedAt.getTime()) / 3_600_000;
  score += Math.min(ageHours * 2, 20);

  return Math.min(score, 100);
}

Reviewer tooling directly affects decision quality and throughput. A good review interface surfaces everything the reviewer needs without requiring navigation: the content itself, the full context (thread, author history, account age), the ML scores and which rules triggered, similar content the reviewer recently decided on, and the action buttons. Keyboard shortcuts and pre-defined decision reasons (not free-text) reduce decision fatigue and produce structured audit logs.

One design principle that gets missed: reviewers should see the ML score only after forming an initial impression when studying bias effects in new label categories. Leading with a 0.87 toxicity score primes the reviewer and inflates inter-rater agreement in a way that inflates model accuracy metrics without capturing genuine human judgment.

Appeal Workflows

DSA Article 17 requires platforms to notify users of content removal with a statement of reasons, and Article 20 requires an internal complaint mechanism. COPPA imposes separate obligations around parental consent and notice for minors’ content.

The appeal flow has three tiers:

1. Self-service appeal: user submits context, system routes back to a human reviewer (ideally not the original reviewer) with the original decision and the appeal text. Most appeals resolve here.
2. Escalation to senior reviewer: if the user disagrees with tier-1 outcome, or if the content falls into a sensitive category (political speech, health information, satire), a senior reviewer with additional policy training reviews.
3. Out-of-court dispute settlement: DSA mandates access to certified out-of-court dispute settlement bodies for EU users. Your system needs to be able to export the full decision history, evidence, and reviewer notes for a given content item.

interface AppealRequest {
  contentId: string;
  userId: string;
  originalDecision: ModerationAction;
  appealText: string;
  submittedAt: Date;
}

interface AppealRecord {
  appealId: string;
  request: AppealRequest;
  tier: 1 | 2 | 3;
  assignedReviewerId: string;
  outcome?: "upheld" | "overturned" | "escalated";
  outcomereasonCode?: string;
  resolvedAt?: Date;
  statementOfReasons: string; // DSA Article 17 compliance
}

async function routeAppeal(
  appeal: AppealRequest,
  originalReviewerId: string,
): Promise<AppealRecord> {
  const appealRecord: AppealRecord = {
    appealId: crypto.randomUUID(),
    request: appeal,
    tier: 1,
    // Assign to different reviewer than original decision
    assignedReviewerId: await assignAppealReviewer({
      excludeReviewerId: originalReviewerId,
      contentCategory: appeal.originalDecision.reason,
    }),
    statementOfReasons: generateStatementOfReasons(appeal.originalDecision),
  };

  await appealQueue.enqueue(appealRecord);
  await notifyUser(appeal.userId, {
    type: "appeal_received",
    appealId: appealRecord.appealId,
    expectedResolutionAt: addBusinessHours(new Date(), 72), // DSA requires timely response
  });

  return appealRecord;
}

The statementOfReasons field is not optional for DSA compliance. It must explain what rule or policy the content violated, referencing the platform’s community standards by specific section, not just a generic “violated our terms of service.”

Feedback Loops and Model Retraining

The human review decisions are your highest-quality training signal. Every reviewer decision should flow back into a feedback pipeline:

• Agreement signal: when a human reviewer confirms an automated removal (high confidence), that is a positive training example for the model.
• Correction signal: when a human reviewer overturns an automated removal, that is a negative training example. These cases are disproportionately valuable.
• Appeal overturn signal: when a tier-1 decision gets overturned at tier-2, that reveals systematic miscalibration.

interface ReviewerDecision {
  contentId: string;
  originalMlResult: ClassificationResult;
  reviewerAction: "confirm_remove" | "restore" | "escalate" | "dismiss_flag";
  reviewerId: string;
  decidedAt: Date;
  contextNotes?: string;
}

async function ingestReviewerDecision(decision: ReviewerDecision): Promise<void> {
  const isModelCorrect =
    (decision.reviewerAction === "confirm_remove" &&
      decision.originalMlResult.label !== "clean") ||
    (decision.reviewerAction === "restore" &&
      decision.originalMlResult.label === "clean");

  const trainingExample: TrainingExample = {
    contentId: decision.contentId,
    label: decision.reviewerAction === "restore" ? "clean" : decision.originalMlResult.label,
    source: "human_review",
    reviewerId: decision.reviewerId,
    modelVersion: decision.originalMlResult.modelVersion,
    originalConfidence: decision.originalMlResult.confidence,
    isModelCorrection: !isModelCorrect,
    createdAt: decision.decidedAt,
  };

  await trainingPipeline.addExample(trainingExample);

  // Trigger threshold recalibration check if correction rate spikes
  const recentCorrectionRate = await metrics.getRecentCorrectionRate({
    windowHours: 24,
    modelVersion: decision.originalMlResult.modelVersion,
  });

  if (recentCorrectionRate > 0.15) {
    await alertOncall({
      severity: "warning",
      message: `Model ${decision.originalMlResult.modelVersion} correction rate ${recentCorrectionRate.toFixed(2)} exceeds 0.15 threshold`,
    });
  }
}

Batch retraining against accumulated human-labeled examples should run on a cadence that matches your content velocity. A platform with millions of daily posts might retrain weekly; a smaller platform quarterly. The critical discipline is versioning every deployed model and tracking per-version precision, recall, and false positive rates continuously in production. Without this, model drift is invisible until a PR crisis or a regulatory audit.

Tradeoffs

Decision	Option A	Option B	When to choose A
Classification latency	Sync before publish (blocks submission)	Async after publish (content briefly live)	NSFW or CSAM on media platforms where exposure risk is high
Confidence thresholds	Single threshold (publish or remove)	Three-tier (auto-block / review / pass)	Almost always — single threshold maximizes false positives or false negatives
ML hosting	Managed API (Perspective, Rekognition)	Self-hosted model	Self-host when volume exceeds ~$50K/yr in API costs or when data residency requires it
Reviewer queue	Single shared queue	Per-category specialized queues	Specialized queues for platforms with distinct content types (image vs text vs video) where reviewer expertise differs
Training data sourcing	Human review decisions only	Human review + third-party labeled datasets	Combined approach — internal labels capture your platform’s unique distribution; external datasets prevent cold start
Appeal routing	Same reviewer tier	Fresh reviewer, independent of original	Fresh reviewer always — reduces confirmation bias and satisfies DSA independence expectations

Production Considerations

False positive rates compound with volume. A 1% false positive rate sounds acceptable until you are at 10 million posts per day: 100,000 legitimate pieces of content wrongly removed. False positives disproportionately affect minority communities whose vernacular and cultural references are underrepresented in training data. Measure false positive rates segmented by language, content category, and author demographics where data allows.

Reviewer burnout is a system design problem, not an HR problem. Sustained exposure to harmful content at high volume causes measurable psychological harm. The architecture should enforce daily exposure limits (not just soft recommendations), rotate reviewers across content categories so no one stays on CSAM or graphic violence exclusively, and build mandatory break intervals into the queue assignment logic. Platforms that ignore this face turnover that destroys the institutional knowledge embedded in their human review layer.

Legal compliance is structural, not procedural. DSA Article 22 requires Very Large Online Platforms (VLOPs: 45M+ EU monthly active users) to conduct annual systemic risk assessments and submit to independent audits. Article 17’s statement-of-reasons obligation applies to all platforms. COPPA’s verifiable parental consent requirements apply to any platform “directed to children” or with actual knowledge of under-13 users. These obligations are not checkbox exercises: they require immutable audit logs, decision provenance, and exportable records. Design your data model to produce these from day one rather than retrofitting.

Adversarial robustness degrades continuously. Users who want to evade moderation will find a way: l33t-speak character substitutions, image overlays that confuse classifiers, multi-language obfuscation, coordinated inauthentic behavior that looks organic until you see the network graph. Your pre-publication filter rules need a curation process that treats them as living code: version-controlled, peer-reviewed, and updated on a defined cadence. Red-team your own classifiers periodically with adversarial examples generated by your own team.

Threshold calibration is a continuous job. The confidence thresholds you set at launch will be wrong within six months because user behavior evolves and model drift occurs. Instrument your pipeline to track, per model version: precision at each threshold, recall at each threshold, human reviewer overturn rate, and appeal overturn rate. Build dashboards that make drift visible before it becomes a crisis.

Closing

A content moderation system is not a feature you ship once. It is an operational subsystem with its own data pipelines, model lifecycle, reviewer workforce, legal compliance obligations, and feedback loops. The architecture described here separates the concerns cleanly: fast deterministic filters catch the known-bad early, ML classification handles the probabilistic middle, human reviewers resolve the uncertain cases, and appeal workflows satisfy legal obligations while surfacing model calibration errors. The feedback loop closes when reviewer decisions flow back into training data. Without that loop, every other layer decays.

The hardest part is not the ML pipeline. It is the organizational commitment to calibrate thresholds empirically, protect reviewers from sustained harm, and treat false positives against real users as a first-class engineering metric alongside recall.