Designing a Distributed Configuration Service: Dynamic Toggles, Hierarchical Overrides, and Consistency at Scale

Every team starts with environment variables. Then someone needs per-environment overrides. Then feature toggles that can be flipped without a deploy. Then service-specific overrides that should not affect other services. Then an audit trail after a config change takes down production at 2am.

By the time you have added the fourth layer of override logic to your .env files and Kubernetes secrets, you have built a broken configuration system without a coherent design. A configuration service is one of those pieces of infrastructure that looks simple until you actually run it in production.

This article covers what a distributed configuration service needs to do, how to model it, the consistency and propagation tradeoffs, and the operational problems that will catch you if you skip them.

What Configuration Services Actually Need to Solve

The core problem is not storage. Any database can store key-value pairs. The hard parts are:

• Hierarchy: a value at the service level should override the environment level, which overrides the global default. Not every resolution order is the same.
• Dynamic updates: config consumers need to pick up changes without restarting. In a microservices architecture, restarting 30 services to propagate a config change is not viable.
• Consistency guarantees: when you flip a toggle, do all instances see it at the same time? If not, what are the consequences?
• Versioning and rollback: the ability to revert a bad config change in under 30 seconds.
• Audit trail: who changed what, when, and why. Required for postmortems and, increasingly, compliance.
• Config drift detection: noticing when the running config of a service has diverged from what the configuration service says it should be.

These are not separate features you bolt on later. They are design constraints that affect the data model from the beginning.

The Hierarchy Model

A flat key-value store cannot represent config inheritance cleanly. You need a layered model where more-specific scopes override less-specific ones.

type ConfigScope =
  | { type: "global" }
  | { type: "environment"; environment: string }
  | { type: "service"; environment: string; service: string }
  | { type: "instance"; environment: string; service: string; instanceId: string };

interface ConfigEntry {
  key: string;
  value: string;
  scope: ConfigScope;
  version: number;
  createdAt: Date;
  updatedAt: Date;
  updatedBy: string;
  comment?: string;
}

Resolution works by starting at the most specific scope and walking up until a value is found. The resolution order is: instance > service > environment > global.

function resolveConfig(
  key: string,
  context: { environment: string; service: string; instanceId: string },
  store: ConfigStore
): string | undefined {
  const scopes: ConfigScope[] = [
    { type: "instance", environment: context.environment, service: context.service, instanceId: context.instanceId },
    { type: "service", environment: context.environment, service: context.service },
    { type: "environment", environment: context.environment },
    { type: "global" },
  ];

  for (const scope of scopes) {
    const entry = store.get(key, scope);
    if (entry !== undefined) {
      return entry.value;
    }
  }

  return undefined;
}

The version field on each entry is not optional. It is what makes rollbacks deterministic and what the audit trail references.

Consistency Models

This is where most teams make a mistake by not thinking explicitly about what they need.

Eventual Consistency

With eventual consistency, config changes propagate to all consumers eventually, but not simultaneously. A change you push at T=0 might reach instance A at T=100ms and instance B at T=2s. During the propagation window, different instances of the same service are running with different config.

For most config changes this is fine. If you are changing a log level or a timeout value, a 2-second window of inconsistency has no user-visible consequence.

For feature toggles that gate revenue paths or modify data schemas, eventual consistency is a liability. If some instances are running with the toggle on and others with it off, you can corrupt state or create race conditions that are difficult to debug.

Strong Consistency

With strong consistency, the config service guarantees that all readers see a change at the same time, or the change is not visible to any reader. This requires consensus across the config service’s own replicas (Raft, Paxos, or equivalent) and a change notification protocol that confirms delivery before marking a change as active.

The cost is latency on writes and more complex infrastructure. For most teams with under 1000 service instances, the operational cost of strong consistency is worth it for any config that affects correctness.

A practical middle ground: classify your config keys by consistency requirement.

type ConsistencyClass = "relaxed" | "strict";

interface ConfigKeyMetadata {
  key: string;
  consistencyClass: ConsistencyClass;
  description: string;
}

// "relaxed" keys: log level, timeouts, non-critical feature toggles
// "strict" keys: payment feature toggles, schema migrations flags, auth config

Only route strict-class keys through the strong-consistency path. Everything else can use the eventual-consistency path with a shorter TTL on cached values.

Change Propagation: Push vs Poll

Services learn about config changes in one of two ways: the config service pushes changes to them, or they poll on an interval.

Polling

Polling is simple to implement and resilient to config service outages. If the config service goes down, services keep running with their cached config. The downside is staleness proportional to the poll interval. A 30-second poll interval means a critical toggle flip can take up to 30 seconds to take effect.

class PollingConfigClient {
  private cache = new Map<string, string>();
  private version = 0;

  constructor(
    private readonly serviceUrl: string,
    private readonly context: ConfigContext,
    private readonly intervalMs: number = 30_000
  ) {}

  start(): void {
    this.fetchAndUpdate();
    setInterval(() => this.fetchAndUpdate(), this.intervalMs);
  }

  private async fetchAndUpdate(): Promise<void> {
    try {
      const response = await fetch(
        `${this.serviceUrl}/config?since=${this.version}&env=${this.context.environment}&service=${this.context.service}`
      );
      const { entries, latestVersion } = await response.json();

      for (const entry of entries) {
        this.cache.set(entry.key, entry.value);
      }
      this.version = latestVersion;
    } catch {
      // Keep serving stale cache on service error
    }
  }

  get(key: string): string | undefined {
    return this.cache.get(key);
  }
}

The since version parameter is important: clients should only fetch deltas, not the full config on every poll. At 1000 service instances polling every 30 seconds, full-config fetches become a significant read load on the config service.

Push via Server-Sent Events

Server-Sent Events give you near-real-time propagation with a simple implementation. The config service streams change events to connected clients.

// Config service: SSE endpoint
app.get("/config/stream", (req, res) => {
  const { environment, service } = req.query as ConfigContext;

  res.setHeader("Content-Type", "text/event-stream");
  res.setHeader("Cache-Control", "no-cache");
  res.setHeader("Connection", "keep-alive");

  const unsubscribe = changeNotifier.subscribe(environment, service, (event) => {
    res.write(`data: ${JSON.stringify(event)}\n\n`);
  });

  req.on("close", () => {
    unsubscribe();
  });
});

// Client: connecting to the stream
class StreamingConfigClient {
  private cache = new Map<string, string>();

  connect(serviceUrl: string, context: ConfigContext): void {
    const url = `${serviceUrl}/config/stream?environment=${context.environment}&service=${context.service}`;
    const eventSource = new EventSource(url);

    eventSource.onmessage = (event) => {
      const change: ConfigChangeEvent = JSON.parse(event.data);
      this.cache.set(change.key, change.value);
    };

    eventSource.onerror = () => {
      // EventSource reconnects automatically; clients continue serving cache
    };
  }
}

Push is more operationally complex because the config service now holds long-lived connections. At 1000 instances, this is 1000 open connections. That is not a problem for a modern Go or Node process, but it means your config service cannot be scaled behind a standard stateless load balancer without sticky sessions or a pub/sub fanout layer.

A common architecture: use Redis Pub/Sub or Kafka as the fanout layer between config service nodes and client connections, so any config service instance can handle any client’s stream.

Versioning and Rollback

Every write to the config store creates a new version, and the old version must be retained. This is not optional if you want rollbacks that work under pressure.

interface ConfigVersion {
  key: string;
  scope: ConfigScope;
  value: string;
  version: number;
  previousVersion: number | null;
  changedBy: string;
  changedAt: Date;
  comment: string;
}

async function rollback(
  key: string,
  scope: ConfigScope,
  targetVersion: number,
  initiatedBy: string,
  store: ConfigVersionStore
): Promise<void> {
  const targetEntry = await store.getVersion(key, scope, targetVersion);
  if (!targetEntry) {
    throw new Error(`Version ${targetVersion} not found for key ${key}`);
  }

  await store.write({
    key,
    scope,
    value: targetEntry.value,
    changedBy: initiatedBy,
    comment: `Rollback to version ${targetVersion}`,
  });
}

Rollback is a new write, not a deletion of the current entry. This preserves the full history and means the audit log accurately reflects that a rollback happened, who triggered it, and when.

Set a retention policy. Most teams keep 90 days of config history. Anything older than that is rarely needed for debugging and starts to accumulate storage cost.

The Audit Trail

The audit trail is the thing teams skip until they are in a postmortem and cannot reconstruct what changed. It needs to answer two questions: “what is different between what is running and what was running before the incident?” and “who made the change?”

interface AuditEvent {
  id: string;
  timestamp: Date;
  action: "create" | "update" | "delete" | "rollback";
  key: string;
  scope: ConfigScope;
  previousValue: string | null;
  newValue: string;
  changedBy: string;
  changedByEmail: string;
  comment: string;
  sourceIp: string;
  correlationId: string;
}

Audit events should be written to an append-only store, separate from the config store itself. If you are storing config in PostgreSQL, write audit events to a separate table with no delete permissions granted to the application user. If you are storing config in a managed key-value service, write audit events to an immutable log (CloudTrail, BigQuery, S3 with object lock).

Config Drift

Config drift is the gap between what the config service says a service should have and what the service is actually running with. It happens for several reasons:

• A service cached config at startup and the push/poll mechanism silently failed.
• A developer directly edited a Kubernetes secret or environment variable without going through the config service.
• A config service node went down and a subset of services did not receive a change.

Detecting drift requires services to periodically report their running config fingerprint back to the config service.

// Service side: report running config fingerprint
async function reportConfigFingerprint(
  client: ConfigReportingClient,
  runningConfig: Map<string, string>,
  context: ConfigContext
): Promise<void> {
  const fingerprint = computeFingerprint(runningConfig);
  await client.report({
    environment: context.environment,
    service: context.service,
    instanceId: context.instanceId,
    fingerprint,
    reportedAt: new Date(),
  });
}

function computeFingerprint(config: Map<string, string>): string {
  const sorted = [...config.entries()].sort(([a], [b]) => a.localeCompare(b));
  return createHash("sha256").update(JSON.stringify(sorted)).digest("hex");
}

The config service computes the expected fingerprint for each service instance and alerts when the reported fingerprint diverges. This gives you a drift detection SLA: if a service has not reported a matching fingerprint within N minutes of a config change, page someone.

Tradeoffs

Dimension	Eventual Consistency	Strong Consistency
Write latency	Low (async propagation)	Higher (requires quorum)
Read latency	Low (local cache)	Low with local cache, higher on cache miss
Propagation window	Seconds to minutes	Near-zero (push) or bounded by poll interval
Infrastructure complexity	Low	Higher (needs consensus layer)
Safe for correctness-critical config	No	Yes

Dimension	Polling	Push (SSE/WebSocket)
Implementation complexity	Low	Medium
Config service statefulness	Stateless	Stateful (open connections)
Propagation latency	Bounded by interval	Near-real-time
Resilience to config service outage	High (serves stale cache)	High (clients buffer and reconnect)
Load on config service	Predictable	Scales with connection count
Works behind standard load balancer	Yes	Requires sticky sessions or pub/sub fanout

Production Considerations

Bootstrap problem: services need config before they have connected to the config service. Always ship a bundled default config file in the service image. The config service provides overrides, not the sole source of truth at startup.

Secret separation: do not mix secrets (database passwords, API keys) with operational config (feature toggles, timeouts) in the same store. Secrets need different access controls, different audit requirements, and different rotation workflows. Keep them in a dedicated secrets manager.

Config validation at write time: reject invalid values before they reach services. A config service that accepts any string for a value that should be an integer will eventually cause a hard-to-debug runtime error. Define schemas for config keys.

interface ConfigKeySchema {
  key: string;
  type: "string" | "integer" | "float" | "boolean" | "json";
  required: boolean;
  allowedValues?: string[];
  minValue?: number;
  maxValue?: number;
  validate(value: string): ValidationResult;
}

Gradual rollout via config: feature toggles that affect a percentage of traffic should live in the config service with a consistent hashing strategy (hash on user ID, not random) so a user gets the same experience across requests. If you push this logic into the config service’s evaluation layer, services do not need to implement it themselves.

Circuit breaking on config service: if a service cannot reach the config service, it should continue running with its last known good config, not fail. Hard-coupling service health to config service availability means a single config service outage takes down your entire infrastructure.

A well-designed configuration service reduces the blast radius of bad config changes from “restart 30 services to roll back” to “flip a value and watch propagation metrics confirm delivery.” The audit trail, versioning, and drift detection are what make that confidence real rather than assumed. Config management is infrastructure, and it deserves the same engineering investment as your database or message queue.