System Design ·

Designing a Ride-Sharing Backend: Real-Time Matching, ETA Prediction, and Dynamic Pricing at Scale

A deep-dive into the backend architecture of a ride-sharing platform covering geospatial driver-rider matching with geohashing, graph-based ETA prediction with real-time traffic adjustment, surge pricing mechanics, and the distributed systems challenges behind high-throughput location streaming and state consistency.

Designing a Ride-Sharing Backend: Real-Time Matching, ETA Prediction, and Dynamic Pricing at Scale

Ride-sharing looks like a simple dispatch problem until you run the numbers. A platform with 200,000 active drivers and 50,000 concurrent passengers generates roughly 50,000 location updates per second. Every ride request must match a driver within hundreds of milliseconds. ETA predictions need to account for real-time traffic and update continuously as conditions change. Pricing must respond to demand in seconds, not minutes.

The interesting engineering is not in any single component. It is in how the matching, routing, pricing, and streaming layers interact under load without producing inconsistent state or stale decisions. This article walks through the backend architecture required to make those layers work together.

The Matching Problem

Driver-rider matching is a two-sided search with a hard latency constraint. When a passenger requests a ride, you need to find the best available driver nearby before the passenger decides to open a competitor’s app. “Nearby” means within a configurable radius that varies by market density. “Best” is a score over multiple dimensions: proximity, driver rating, vehicle type, acceptance rate history.

Geohash-Based Candidate Selection

Scanning all active drivers for every ride request does not scale. At 200,000 active drivers, even a fast in-memory scan adds up when you have thousands of concurrent requests. The standard approach is to use a geospatial index to narrow the candidate set before scoring.

Redis’s geospatial commands (GEOADD, GEOSEARCH) store member positions as geohash-encoded sorted set scores internally, giving you O(N+log(M)) proximity queries without maintaining a separate index.

interface DriverStatus {
  driverId: string;
  latitude: number;
  longitude: number;
  vehicleType: "economy" | "comfort" | "xl";
  rating: number;
  acceptanceRate: number;
  isAvailable: boolean;
}

interface RideRequest {
  requestId: string;
  passengerId: string;
  pickupLatitude: number;
  pickupLongitude: number;
  vehicleType: "economy" | "comfort" | "xl";
  requestedAt: number;
}

interface MatchCandidate {
  driverId: string;
  distanceMeters: number;
  etaSeconds: number;
  score: number;
}

async function findCandidateDrivers(
  request: RideRequest,
  radiusMeters: number,
  limit: number
): Promise<Array<{ driverId: string; distanceMeters: number }>> {
  const geoKey = `drivers:available:${request.vehicleType}`;

  // Redis GEOSEARCH returns members sorted by distance ascending
  const results = await redis.geosearch(
    geoKey,
    "FROMLONLAT",
    request.pickupLongitude,
    request.pickupLatitude,
    "BYRADIUS",
    radiusMeters / 1000,
    "km",
    "ASC",
    "COUNT",
    limit,
    "WITHCOORD",
    "WITHDIST"
  );

  return results.map((r: any) => ({
    driverId: r[0],
    distanceMeters: parseFloat(r[1]) * 1000,
  }));
}

The key partitioning decision here is separating the geo index by vehicle type. drivers:available:economy and drivers:available:xl are separate keys. This avoids the overhead of post-filtering a combined set by vehicle type and keeps each key smaller.

Scoring and Selection

Proximity is the dominant factor, but raw distance produces poor matches in practice. A driver 800m away who rejects 40% of requests is worse than one 1.2km away with a 95% acceptance rate. The matching score should fold in multiple signals.

interface DriverMetadata {
  rating: number;
  acceptanceRate: number;
  completionRate: number;
  onlineMinutesToday: number;
}

function computeMatchScore(
  candidate: { driverId: string; distanceMeters: number },
  metadata: DriverMetadata,
  etaSeconds: number
): number {
  // Normalize each dimension to [0, 1] range
  const distanceScore = Math.max(0, 1 - candidate.distanceMeters / 5000);
  const etaScore = Math.max(0, 1 - etaSeconds / 600); // 600s = 10 min cap
  const ratingScore = (metadata.rating - 1) / 4; // scale 1-5 to 0-1
  const acceptanceScore = metadata.acceptanceRate;

  // Weights reflect product priorities:
  // ETA matters most (passenger waits for this), then distance as a proxy,
  // then reliability signals
  return (
    etaScore * 0.40 +
    distanceScore * 0.30 +
    ratingScore * 0.15 +
    acceptanceScore * 0.15
  );
}

async function matchDriver(request: RideRequest): Promise<string | null> {
  const candidates = await findCandidateDrivers(request, 5000, 20);

  if (candidates.length === 0) return null;

  const scored: MatchCandidate[] = await Promise.all(
    candidates.map(async (c) => {
      const [metadata, etaSeconds] = await Promise.all([
        getDriverMetadata(c.driverId),
        estimatePickupEta(c.driverId, request.pickupLatitude, request.pickupLongitude),
      ]);

      return {
        driverId: c.driverId,
        distanceMeters: c.distanceMeters,
        etaSeconds,
        score: computeMatchScore(c, metadata, etaSeconds),
      };
    })
  );

  scored.sort((a, b) => b.score - a.score);
  return scored[0].driverId;
}

The Promise.all across candidates is important. Fetching ETA sequentially for 20 candidates adds up; parallel resolution keeps the total matching latency bounded to approximately one ETA call latency plus overhead.

Offer Dispatch and Race Conditions

After selecting the best candidate, you send an offer to the driver. But the driver might be offline, might decline, or might have been matched to another ride in the milliseconds since your query ran. You need to handle all three outcomes.

const OFFER_TIMEOUT_MS = 15_000;
const MAX_RETRIES = 3;

async function dispatchWithFallback(
  request: RideRequest
): Promise<{ driverId: string; accepted: boolean } | null> {
  for (let attempt = 0; attempt < MAX_RETRIES; attempt++) {
    const driverId = await matchDriver(request);
    if (!driverId) return null; // no drivers available in radius

    // Atomically claim the driver using a Redis lock
    // Key: driver:<id>:lock, TTL matches offer timeout
    const lockAcquired = await redis.set(
      `driver:${driverId}:lock`,
      request.requestId,
      "NX",
      "PX",
      OFFER_TIMEOUT_MS
    );

    if (!lockAcquired) {
      // Driver was claimed by another request; retry with next candidate
      continue;
    }

    const accepted = await sendOfferAndWait(driverId, request, OFFER_TIMEOUT_MS);

    if (!accepted) {
      await redis.del(`driver:${driverId}:lock`);
      // Mark driver as temporarily deprioritized to avoid repeatedly
      // offering the same non-responsive driver in rapid succession
      await redis.setex(`driver:${driverId}:cooldown`, 30, "1");
      continue;
    }

    return { driverId, accepted: true };
  }

  return null;
}

The Redis SET NX PX pattern provides a lightweight distributed lock for the offer window. This is not a substitute for a proper distributed lock service if you need strict guarantees, but for ride dispatch the semantics are acceptable: if the lock expires before acceptance, the request retries. Duplicate acceptance is prevented by a separate idempotency check when the driver’s response arrives.

ETA Prediction

ETA has two components: the time to pick up the passenger (pickup ETA), and the time from pickup to destination (trip ETA). Both require routing over a road network, not straight-line distance.

Graph-Based Routing

The road network is modeled as a directed weighted graph where nodes are road intersections and edges are road segments. Edge weights represent travel time, not distance. A preprocessing step (Contraction Hierarchies or similar) speeds up Dijkstra queries from O(V log V) to milliseconds on city-scale graphs.

interface RoadNode {
  nodeId: number;
  latitude: number;
  longitude: number;
}

interface RoadEdge {
  fromNodeId: number;
  toNodeId: number;
  baseTimeSeconds: number;  // historical average travel time
  currentTimeSeconds: number; // adjusted for real-time traffic
  distanceMeters: number;
  roadClass: "motorway" | "primary" | "secondary" | "residential";
}

interface RouteResult {
  durationSeconds: number;
  distanceMeters: number;
  polyline: Array<[number, number]>;
}

function dijkstra(
  graph: Map<number, RoadEdge[]>,
  sourceNodeId: number,
  targetNodeId: number
): RouteResult | null {
  const dist = new Map<number, number>();
  const prev = new Map<number, number>();
  // Min-heap keyed by cost: [cost, nodeId]
  const heap: Array<[number, number]> = [[0, sourceNodeId]];
  dist.set(sourceNodeId, 0);

  while (heap.length > 0) {
    heap.sort((a, b) => a[0] - b[0]); // use a real min-heap in production
    const [cost, nodeId] = heap.shift()!;

    if (nodeId === targetNodeId) {
      return reconstructRoute(dist, prev, sourceNodeId, targetNodeId, graph);
    }

    if (cost > (dist.get(nodeId) ?? Infinity)) continue;

    for (const edge of graph.get(nodeId) ?? []) {
      const newCost = cost + edge.currentTimeSeconds;
      if (newCost < (dist.get(edge.toNodeId) ?? Infinity)) {
        dist.set(edge.toNodeId, newCost);
        prev.set(edge.toNodeId, nodeId);
        heap.push([newCost, edge.toNodeId]);
      }
    }
  }

  return null;
}

In production you would not implement this from scratch. OSRM, Valhalla, or GraphHopper are mature open-source routing engines that handle the graph preprocessing and query acceleration. The point is that the edge weights must use currentTimeSeconds, not baseTimeSeconds, for traffic-adjusted ETA.

Real-Time Traffic Adjustment

Traffic data arrives from multiple sources: GPS probes (anonymized location streams from active drivers), historical speed data per road segment per time-of-day, and commercial traffic feeds (HERE, Google Maps Platform). The routing engine needs these weights updated continuously.

interface TrafficUpdate {
  edgeId: string;
  fromNodeId: number;
  toNodeId: number;
  observedSpeedKph: number;
  sampleCount: number;
  observedAt: number;
}

interface EdgeSpeedModel {
  historicalSpeedKph: number;
  currentSpeedKph: number;
  confidence: number; // 0-1, based on sample count recency
  lastUpdated: number;
}

function computeCurrentEdgeTime(
  edge: RoadEdge,
  model: EdgeSpeedModel
): number {
  // Blend real-time observation with historical baseline
  // Low confidence (few samples) defaults toward historical
  const blendedSpeed =
    model.confidence * model.currentSpeedKph +
    (1 - model.confidence) * model.historicalSpeedKph;

  return (edge.distanceMeters / 1000 / blendedSpeed) * 3600;
}

async function ingestTrafficProbe(update: TrafficUpdate): Promise<void> {
  const key = `traffic:edge:${update.edgeId}`;
  const existing = await redis.hgetall(key) as Partial<Record<string, string>>;

  const prevSpeed = parseFloat(existing.currentSpeedKph ?? "0");
  const prevSamples = parseInt(existing.sampleCount ?? "0", 10);

  // Exponential moving average weighted by sample count
  const newSpeed =
    prevSamples > 0
      ? prevSpeed * 0.7 + update.observedSpeedKph * 0.3
      : update.observedSpeedKph;

  await redis.hset(key, {
    currentSpeedKph: newSpeed.toFixed(2),
    sampleCount: prevSamples + update.sampleCount,
    lastUpdated: update.observedAt,
  });

  await redis.expire(key, 300); // discard stale observations after 5 minutes
}

The 5-minute TTL is intentional. Traffic conditions older than 5 minutes are often more misleading than helpful. If fresh data stops arriving for a segment, the routing engine falls back to the historical baseline, which is preferable to trusting stale real-time data.

ETA Confidence Intervals

Presenting a single number as ETA overstates certainty. A better interface exposes a range.

interface EtaEstimate {
  optimisticSeconds: number;  // p10 (light traffic)
  expectedSeconds: number;    // p50 (current conditions)
  pessimisticSeconds: number; // p90 (heavy traffic)
  confidenceScore: number;    // 0-1
}

function buildEtaEstimate(
  baseRoute: RouteResult,
  historicalVariance: number, // stddev in seconds for this route and time of day
  trafficConfidence: number
): EtaEstimate {
  const spread = historicalVariance * (1 + (1 - trafficConfidence));

  return {
    optimisticSeconds: Math.round(baseRoute.durationSeconds - spread * 0.5),
    expectedSeconds: baseRoute.durationSeconds,
    pessimisticSeconds: Math.round(baseRoute.durationSeconds + spread * 1.5),
    confidenceScore: trafficConfidence,
  };
}

Passengers shown “8-12 minutes” tolerate uncertainty better than those shown “10 minutes” and seeing it extend to 14. The p90 case is the psychologically significant one.

Dynamic Pricing

Surge pricing adjusts fares upward when demand exceeds supply in a geographic area. The mechanics involve three decisions: how to define zones, how to compute the supply-demand ratio, and how to map that ratio to a multiplier.

Zone Definition

H3 (Uber’s open-source hexagonal grid library) is the standard approach. It divides the world into hexagonal cells at configurable resolutions. Hexagons tile uniformly without the edge distortion you get from rectangular grids, which matters for fair pricing near zone boundaries.

// H3 resolution 8 cells are ~460m edge-to-edge, appropriate for urban surge zones
const SURGE_ZONE_RESOLUTION = 8;

interface ZoneState {
  h3Index: string;
  activeRequests: number;
  availableDrivers: number;
  demandSupplyRatio: number;
  surgeMultiplier: number;
  computedAt: number;
}

function computeSurgeMultiplier(demandSupplyRatio: number): number {
  // Stepwise multiplier prevents jarring jumps
  // Multipliers are capped to avoid regulatory and PR issues
  if (demandSupplyRatio < 1.2) return 1.0;
  if (demandSupplyRatio < 1.5) return 1.2;
  if (demandSupplyRatio < 2.0) return 1.5;
  if (demandSupplyRatio < 2.5) return 1.8;
  if (demandSupplyRatio < 3.0) return 2.0;
  return 2.5; // hard cap
}

async function recomputeZoneSurge(h3Index: string): Promise<ZoneState> {
  const [activeRequests, availableDrivers] = await Promise.all([
    countPendingRequestsInZone(h3Index),
    countAvailableDriversInZone(h3Index),
  ]);

  // Avoid division by zero; treat zero drivers as extreme scarcity
  const ratio =
    availableDrivers === 0
      ? activeRequests > 0 ? 5.0 : 1.0
      : activeRequests / availableDrivers;

  const state: ZoneState = {
    h3Index,
    activeRequests,
    availableDrivers,
    demandSupplyRatio: ratio,
    surgeMultiplier: computeSurgeMultiplier(ratio),
    computedAt: Date.now(),
  };

  await redis.setex(
    `surge:zone:${h3Index}`,
    30,
    JSON.stringify(state)
  );

  return state;
}

Demand Smoothing

Raw demand counts are noisy. A single user rapidly requesting and cancelling rides inflates demand artificially. Smooth the input before computing the ratio.

async function getSmoothedDemand(h3Index: string): Promise<number> {
  // Fetch request counts for the last 3 time windows (each 30s)
  const keys = [0, 30, 60].map(
    (offsetSecs) =>
      `demand:${h3Index}:${Math.floor((Date.now() - offsetSecs * 1000) / 30_000)}`
  );

  const counts = await redis.mget(...keys);
  const values = counts.map((v) => parseInt(v ?? "0", 10));

  // Weighted average: most recent window weighted 3x, then 2x, then 1x
  return (values[0] * 3 + values[1] * 2 + values[2] * 1) / 6;
}

Price Lock During Booking

Once a passenger sees a surge price and begins the booking flow, that multiplier must be locked for the duration of their session. If the surge drops before they confirm, they should get the lower price. If it increases, they keep the quoted price for some tolerance window (typically 2-3 minutes).

interface PriceLock {
  requestId: string;
  passengerId: string;
  multiplier: number;
  lockedAt: number;
  expiresAt: number;
}

async function lockPrice(
  passengerId: string,
  h3Index: string
): Promise<PriceLock> {
  const zoneState = await getSurgeState(h3Index);
  const lock: PriceLock = {
    requestId: crypto.randomUUID(),
    passengerId,
    multiplier: zoneState.surgeMultiplier,
    lockedAt: Date.now(),
    expiresAt: Date.now() + 180_000, // 3-minute booking window
  };

  await redis.setex(
    `price_lock:${passengerId}`,
    180,
    JSON.stringify(lock)
  );

  return lock;
}

async function getFinalMultiplier(passengerId: string): Promise<number> {
  const raw = await redis.get(`price_lock:${passengerId}`);
  if (!raw) {
    // No lock found; use current zone price
    return 1.0;
  }

  const lock: PriceLock = JSON.parse(raw);
  if (Date.now() > lock.expiresAt) {
    // Lock expired; require passenger to re-enter booking flow
    throw new Error("PRICE_LOCK_EXPIRED");
  }

  return lock.multiplier;
}

Location Streaming at Scale

Drivers update their location every 4-5 seconds while online. At 200,000 active drivers this is 40,000-50,000 writes per second. The write path and the read path have almost nothing in common and should be designed separately.

Write Path: Ingestion and Deduplication

Location updates arrive over persistent WebSocket connections. The ingestion tier’s only job is to validate, deduplicate, and forward; it should not perform matching or pricing logic.

const locationBuffer = new Map<string, {
  latitude: number;
  longitude: number;
  timestamp: number;
  speed?: number;
  heading?: number;
}>();

// Flush buffer every 250ms rather than writing each update individually
// At 50K updates/s this reduces Redis write operations by ~12x
setInterval(async () => {
  if (locationBuffer.size === 0) return;

  const batch = Array.from(locationBuffer.entries());
  locationBuffer.clear();

  const pipeline = redis.pipeline();

  for (const [driverId, loc] of batch) {
    // Update the geospatial index used for matching queries
    pipeline.geoadd("drivers:geo", loc.longitude, loc.latitude, driverId);

    // Publish to pub/sub channel for subscriber fan-out
    // Passengers tracking their driver subscribe to this channel
    pipeline.publish(
      `driver:location:${driverId}`,
      JSON.stringify({ lat: loc.latitude, lon: loc.longitude, ts: loc.timestamp })
    );

    // Store last-known location for recovery after reconnection
    pipeline.hset(`driver:lastloc:${driverId}`, {
      lat: loc.latitude.toFixed(6),
      lon: loc.longitude.toFixed(6),
      ts: loc.timestamp,
    });
    pipeline.expire(`driver:lastloc:${driverId}`, 3600);
  }

  await pipeline.exec();
}, 250);

function bufferDriverLocation(
  driverId: string,
  latitude: number,
  longitude: number,
  timestamp: number
): void {
  // Most recent update wins; older positions for the same driver are irrelevant
  locationBuffer.set(driverId, { latitude, longitude, timestamp });
}

State Consistency: Driver Availability

Driver availability state is one of the hairier consistency problems. A driver transitions between: offline, online-available, dispatched (offer sent), en-route-to-pickup, in-trip, and back to online-available. These transitions must be atomic relative to the matching system.

type DriverState =
  | "offline"
  | "available"
  | "offer_pending"
  | "en_route_pickup"
  | "in_trip";

async function transitionDriverState(
  driverId: string,
  from: DriverState,
  to: DriverState
): Promise<boolean> {
  // Lua script runs atomically on the Redis server
  // No other command can execute between the GET and the SET
  const script = `
    local current = redis.call("GET", KEYS[1])
    if current ~= ARGV[1] then
      return 0
    end
    redis.call("SET", KEYS[1], ARGV[2])
    return 1
  `;

  const result = await redis.eval(
    script,
    1,
    `driver:state:${driverId}`,
    from,
    to
  );

  if (result === 1) {
    // Reflect the new state in the availability geo index
    if (to === "available") {
      const loc = await getDriverLastLocation(driverId);
      if (loc) {
        await redis.geoadd("drivers:geo", loc.longitude, loc.latitude, driverId);
      }
    } else {
      // Remove from matching pool until available again
      await redis.zrem("drivers:geo", driverId);
    }
  }

  return result === 1;
}

The Lua script for the compare-and-swap is critical. Without it, two concurrent requests can both read available, both proceed to offer the driver, and both succeed in updating to offer_pending. The Lua approach guarantees the transition is atomic without acquiring a distributed lock.

Tradeoffs

DimensionSimple approachRobust approachWhen to upgrade
Spatial indexPostGIS ST_DWithin onlyRedis GEOSEARCH for matching hot path, PostGIS for analyticsWrite throughput above 5K/s
Matching candidatesFixed 5km radius, top 10Adaptive radius based on market density, 15-20 candidates scoredLow match rates in sparse markets
ETA computationStraight-line distance haversineOSRM/Valhalla graph routing with traffic weightsAny production deployment
Traffic updatesHistorical speed onlyEMA-blended GPS probe data with TTL-based staleness evictionWhen ETA misses exceed 25%
Surge zonesFixed rectangular gridH3 hexagonal grid, resolution tuned per marketBoundary fairness complaints
Driver state transitionsApplication-level lockingAtomic Lua CAS on RedisAny concurrent dispatch scenario
Location ingestionWrite each update immediately250ms dedup buffer with pipeline flushAbove 5K active drivers

Production Considerations

Cold starts in new markets: When a platform launches in a new city, no historical traffic data exists. ETA predictions will be inaccurate for weeks until probe data accumulates. Launch with conservative uncertainty ranges (show 20-minute estimates as “15-25 min”) and tighten as data improves.

Ghost drivers: A driver who loses connectivity does not immediately go offline. Their last-known position stays in the geo index and they continue appearing in match candidates. Set a TTL on the availability index membership: remove any driver whose last location update is older than 60 seconds from the active pool. The driver’s lastloc key persists for recovery; the matching index entry expires.

async function pruneStaleDrivers(): Promise<void> {
  // Run on a 30-second interval
  const allDrivers = await redis.smembers("drivers:online");
  const staleCutoff = Date.now() - 60_000;

  for (const driverId of allDrivers) {
    const raw = await redis.hget(`driver:lastloc:${driverId}`, "ts");
    const lastTs = parseInt(raw ?? "0", 10);

    if (lastTs < staleCutoff) {
      await redis.zrem("drivers:geo", driverId);
      await redis.srem("drivers:online", driverId);
      await redis.set(`driver:state:${driverId}`, "offline");
    }
  }
}

Pricing during outages: If the surge computation service is unavailable, the fallback must be defined and agreed upon in advance. Serving 1.0x (no surge) during a service outage is a deliberate product decision. The alternative (serving stale surge from a cached value) risks showing passengers different prices than what matches are actually being dispatched at. Define the fallback before launch.

Matching latency budget: The full matching cycle (candidate query + parallel ETA fetch + scoring + offer dispatch) should complete in under 300ms for a good passenger experience. Profile each leg separately. ETA computation is usually the bottleneck; pre-computing ETAs from likely pickup zones during idle periods and caching them with short TTLs reduces p99 latency significantly.

Replay and auditing: Every fare calculation must be auditable. Log the inputs to the surge multiplier computation (zone ID, active requests, available drivers, ratio, multiplier, timestamp) alongside every ride record. When a passenger disputes a price, you need to be able to reconstruct exactly why that multiplier was applied at that moment.

The Core Insight

Ride-sharing backend design is fundamentally about three intersecting real-time state machines: driver availability, ride request lifecycle, and zone pricing. Each state machine needs strong consistency guarantees for the transitions that matter (a driver cannot be offered to two passengers simultaneously; a price lock cannot be silently updated mid-booking). But they also need to be loosely coupled from each other so that a surge computation delay does not stall driver matching.

The architecture that holds up under load separates the write path (location ingestion, state transitions) from the read path (proximity queries, ETA estimates, surge lookup) and uses Redis as the coordination layer for anything requiring sub-millisecond atomic operations. PostgreSQL with PostGIS handles the durable, queryable, analytically useful record of what happened. The two systems are kept consistent asynchronously, with the Redis layer treated as the source of truth for operational decisions and the database as the source of truth for billing and history.

Design the consistency boundaries first. Everything else follows from them.

More in System Design

How Amazon Aurora Works Internally: The Log-Is-the-Database Architecture, Quorum Writes, and Storage-Compute Separation Behind Cloud-Native SQL
System Design ·

How Amazon Aurora Works Internally: The Log-Is-the-Database Architecture, Quorum Writes, and Storage-Compute Separation Behind Cloud-Native SQL

A deep dive into Aurora's storage-compute separation, the log-is-the-database design that pushes redo log processing to storage nodes, quorum-based replication across 6 copies in 3 AZs, protection group architecture, fast cloning, Serverless v2 scaling mechanics, and honest tradeoffs versus RDS, self-managed PostgreSQL, CockroachDB, and Cloud Spanner.

How CockroachDB Works Internally: Distributed SQL, Raft Consensus Per Range, and the Architecture Behind Serializable Transactions at Global Scale
System Design ·

How CockroachDB Works Internally: Distributed SQL, Raft Consensus Per Range, and the Architecture Behind Serializable Transactions at Global Scale

A deep dive into CockroachDB's internals: the 512MB range-based data model with automatic splitting, per-range Raft replication, MVCC timestamp ordering with hybrid logical clocks, DistSQL physical planning, serializable transactions with timestamp refreshes, closed timestamps for follower reads, online schema changes using multi-version schema descriptors, and production considerations for hotspots and write amplification.

How Container Runtimes Work Internally: Namespaces, Cgroups, and the OCI Stack From Docker Run to Process Isolation
System Design ·

How Container Runtimes Work Internally: Namespaces, Cgroups, and the OCI Stack From Docker Run to Process Isolation

Containers are not virtual machines and they are not magic. They are a thin composition of Linux kernel primitives: namespaces, cgroups, and a layered filesystem. This article traces exactly what happens between docker run and a running process, covering the OCI spec, containerd, runc, overlay filesystems, and container networking at the kernel level.

How YugabyteDB Works Internally: DocDB Storage, Tablet Splitting, and the Dual API Architecture That Scales PostgreSQL Horizontally
System Design ·

How YugabyteDB Works Internally: DocDB Storage, Tablet Splitting, and the Dual API Architecture That Scales PostgreSQL Horizontally

A deep dive into YugabyteDB internals covering the DocDB storage engine built on RocksDB LSM trees, per-tablet Raft consensus, YSQL and YCQL dual query layers, distributed MVCC with hybrid logical clocks, automatic tablet splitting and rebalancing, xCluster multi-region replication, and production considerations for hotspots, connection pooling, and schema design.