Wide Events and Observability 2.0: Why Structured Events Are Replacing the Three-Pillar Model

The three-pillar model of observability, metrics, logs, and traces, feels natural because it maps to how monitoring tools evolved. Metrics came first (Graphite, StatsD, Prometheus). Logs have always existed. Distributed tracing arrived later (Jaeger, Zipkin, then OpenTelemetry). Most teams adopted each pillar as a separate system, and the observability industry sold them all three stacks as a bundle.

The problem is that these three pillars were designed independently, stored independently, and queried independently. When a production incident requires understanding what happened to a specific user’s request across five services, you open three different UIs, manually correlate timestamps and request IDs, and hope someone set the trace ID correctly in the log context.

Wide events are a different approach. One structured event per unit of work, with dozens or hundreds of high-cardinality attributes attached. No separate systems to correlate. Query anything after the fact.

What Is Wrong With the Three-Pillar Model

The correlation problem is the most immediate pain. You have a trace showing a slow database query, a metric showing elevated p99 latency, and a log line showing an error. Are they the same request? Maybe. You need a trace ID to know, and that trace ID has to be threaded through every log call manually. Teams that instrument carefully get this right. Teams under deadline pressure do not.

The cardinality problem is structural. Prometheus and similar metric systems aggregate at write time. When you record a histogram, you choose the labels up front: http_request_duration_seconds{method="POST", status="200", endpoint="/api/orders"}. Every unique combination of label values creates a new time series. Add a user_id label and you have one series per user. At ten thousand users, your Prometheus server falls over. At a million users, this approach is simply unavailable to you.

This is why you cannot answer “what is the p99 latency for requests from premium plan users?” in most metric systems. You would have had to define that dimension at instrumentation time.

The cost problem is compounding. Running Prometheus, a log aggregation platform, and a distributed tracing backend means three separate storage systems, three separate query engines, and three separate ingestion pipelines. Many organizations pay for all three even when the overlap is substantial. Logs and traces in particular store much of the same information.

The Wide Event Model

A wide event is a single structured record that captures everything relevant about one unit of work: an HTTP request, a background job execution, a cron run, a queue message processed. Instead of emitting a counter for request count, a histogram for latency, a log line for the error, and a span for the database call, you emit one event at the end of the request with all of that information as fields.

Stripe published their approach to this under the name “canonical log lines” in 2019. The idea is that every HTTP request produces one log line at the end, containing the request ID, user ID, endpoint, HTTP method, response status, duration, database query count, cache hit/miss, feature flags evaluated, and any other attribute relevant to debugging. Nothing is discarded. Everything is queryable.

Honeycomb built an entire product around this model. Their insight was that if you treat observability data as structured events rather than pre-aggregated metrics, you can answer arbitrary questions about production behavior without having pre-defined which questions you would want to ask.

The shift is from “define your metrics schema at instrumentation time” to “record everything and query it later.” This is only practical if the storage and query layer can handle it. That is where columnar databases come in.

Why Column Stores Make This Work

Traditional row-oriented databases struggle with wide event analytics. A table with two hundred columns where each query touches five of them wastes I/O reading the other one hundred ninety-five.

Columnar databases store each column separately. A query for p99(latency_ms) WHERE plan = 'enterprise' AND endpoint = '/api/search' reads only the latency_ms, plan, and endpoint columns, ignoring everything else. Compression is also dramatically better because similar values are stored together.

ClickHouse is the most common choice for wide event storage at production scale. It handles billions of rows, supports complex aggregations, and responds to analytical queries in sub-second time. Its MergeTree family of table engines handles time-series data efficiently with configurable retention and compression.

DuckDB has become popular for local and development-time analysis. It can query Parquet files directly, meaning you can export events from your production store and run arbitrary SQL locally without standing up infrastructure. For teams that cannot justify ClickHouse’s operational overhead, DuckDB plus S3 Parquet storage is a viable architecture.

Parquet as a storage format deserves mention. It is columnar, compressed, and readable by almost every analytical tool. Storing wide events as Parquet on S3 gives you a durable, cheap archive that can be queried by Athena, DuckDB, Spark, or ClickHouse external tables.

Schema Design for Wide Events

A good wide event schema has a stable core and an extensible attribute set. The core fields are present on every event. The attribute set grows as instrumentation depth increases.

interface WideEvent {
  // identity
  request_id: string;         // UUID, primary correlation key
  trace_id: string;           // OpenTelemetry trace ID if distributed tracing is used
  span_id: string;            // OpenTelemetry span ID

  // timing
  timestamp: string;          // ISO 8601, UTC
  duration_ms: number;        // total request duration

  // HTTP context
  http_method: string;
  http_path: string;          // normalized, no IDs (e.g. /api/users/:id)
  http_status: number;
  http_user_agent: string;

  // identity and tenancy
  user_id: string | null;
  tenant_id: string | null;
  session_id: string | null;

  // outcome
  outcome: "success" | "error" | "timeout" | "rate_limited";
  error_type: string | null;  // e.g. "ValidationError", "DatabaseError"
  error_message: string | null;

  // resource usage
  db_query_count: number;
  db_duration_ms: number;
  cache_hit_count: number;
  cache_miss_count: number;
  external_call_count: number;
  external_call_duration_ms: number;

  // business context
  plan: string | null;        // "free", "pro", "enterprise"
  feature_flags: Record<string, boolean>;

  // cost attribution (useful for AI workloads)
  llm_model: string | null;
  llm_input_tokens: number | null;
  llm_output_tokens: number | null;
  llm_cost_usd: number | null;

  // deployment context
  service_name: string;
  service_version: string;
  environment: string;        // "production", "staging"
  region: string;
}

Several design decisions here are worth explaining.

http_path should be normalized. Store /api/users/:id rather than /api/users/abc-123. Without normalization, every unique user ID creates a distinct path, and grouping queries by endpoint becomes impossible.

outcome is a categorical field, not just http_status. A 200 response that returns an empty result set because the upstream service timed out is a timeout, not a success. The distinction matters for alerting on business-level failures rather than HTTP-level ones.

feature_flags as a structured map means you can slice any metric by which flags were active for that request. This is the killer feature for gradual rollouts: “show me p99 latency for requests where the new_checkout_flow flag was true, compared to false.”

llm_* fields reflect the reality that AI-augmented applications need cost attribution per request. Knowing that a particular endpoint costs $0.004 per call average, and that p99 is $0.08, is essential for product and pricing decisions.

Emitting Wide Events With OpenTelemetry

OpenTelemetry’s log API is the right primitive for wide events. Logs in OTel are structured records attached to a trace context, which gives you the best of both models: the rich structured event plus the ability to correlate with spans when you need distributed context.

import { logs, SeverityNumber } from "@opentelemetry/api-logs";
import { context, trace } from "@opentelemetry/api";

interface WideEventContext {
  request_id: string;
  user_id?: string;
  tenant_id?: string;
  plan?: string;
  feature_flags?: Record<string, boolean>;
}

class WideEventEmitter {
  private readonly logger = logs.getLogger("wide-events", "1.0.0");
  private readonly attributes: Record<string, unknown> = {};
  private readonly startTime: number;

  // counters accumulated during request processing
  private dbQueryCount = 0;
  private dbDurationMs = 0;
  private cacheHitCount = 0;
  private cacheMissCount = 0;
  private externalCallCount = 0;
  private externalCallDurationMs = 0;

  private outcome: "success" | "error" | "timeout" | "rate_limited" = "success";
  private errorType: string | null = null;
  private errorMessage: string | null = null;

  constructor(private readonly ctx: WideEventContext) {
    this.startTime = Date.now();
    this.attributes["request_id"] = ctx.request_id;
    if (ctx.user_id) this.attributes["user_id"] = ctx.user_id;
    if (ctx.tenant_id) this.attributes["tenant_id"] = ctx.tenant_id;
    if (ctx.plan) this.attributes["plan"] = ctx.plan;
    if (ctx.feature_flags) {
      // flatten flags for columnar storage: feature_flag.new_checkout=true
      for (const [flag, value] of Object.entries(ctx.feature_flags)) {
        this.attributes[`feature_flag.${flag}`] = value;
      }
    }
  }

  recordDbQuery(durationMs: number): void {
    this.dbQueryCount += 1;
    this.dbDurationMs += durationMs;
  }

  recordCacheHit(): void {
    this.cacheHitCount += 1;
  }

  recordCacheMiss(): void {
    this.cacheMissCount += 1;
  }

  recordExternalCall(durationMs: number): void {
    this.externalCallCount += 1;
    this.externalCallDurationMs += durationMs;
  }

  recordError(type: string, message: string): void {
    this.outcome = "error";
    this.errorType = type;
    this.errorMessage = message;
  }

  recordTimeout(): void {
    this.outcome = "timeout";
  }

  addAttribute(key: string, value: unknown): void {
    this.attributes[key] = value;
  }

  emit(httpMethod: string, httpPath: string, httpStatus: number): void {
    const durationMs = Date.now() - this.startTime;

    // pull trace and span IDs from the active OTel context
    const activeSpan = trace.getActiveSpan();
    const spanContext = activeSpan?.spanContext();

    this.logger.emit({
      severityNumber:
        this.outcome === "success"
          ? SeverityNumber.INFO
          : SeverityNumber.ERROR,
      severityText: this.outcome === "success" ? "INFO" : "ERROR",
      body: `${httpMethod} ${httpPath} ${httpStatus} ${durationMs}ms`,
      attributes: {
        ...this.attributes,

        // HTTP
        "http.method": httpMethod,
        "http.path": httpPath,
        "http.status_code": httpStatus,

        // timing
        "duration_ms": durationMs,

        // outcome
        "outcome": this.outcome,
        "error.type": this.errorType,
        "error.message": this.errorMessage,

        // resource usage
        "db.query_count": this.dbQueryCount,
        "db.duration_ms": this.dbDurationMs,
        "cache.hit_count": this.cacheHitCount,
        "cache.miss_count": this.cacheMissCount,
        "external.call_count": this.externalCallCount,
        "external.call_duration_ms": this.externalCallDurationMs,

        // deployment
        "service.name": process.env.SERVICE_NAME ?? "unknown",
        "service.version": process.env.SERVICE_VERSION ?? "unknown",
        "deployment.environment": process.env.NODE_ENV ?? "unknown",

        // trace correlation
        "trace_id": spanContext?.traceId,
        "span_id": spanContext?.spanId,
      },
    });
  }
}

The usage pattern is to create one WideEventEmitter at the start of each request and pass it through request context. Every layer of the application calls recordDbQuery, recordCacheHit, and similar methods as work happens. At the end of the request, emit fires once.

import { AsyncLocalStorage } from "node:async_hooks";

const eventStorage = new AsyncLocalStorage<WideEventEmitter>();

// Hono middleware example
app.use("*", async (c, next) => {
  const emitter = new WideEventEmitter({
    request_id: crypto.randomUUID(),
    user_id: c.get("userId") ?? undefined,
    tenant_id: c.get("tenantId") ?? undefined,
    plan: c.get("plan") ?? undefined,
    feature_flags: c.get("featureFlags") ?? undefined,
  });

  await eventStorage.run(emitter, async () => {
    await next();
    emitter.emit(
      c.req.method,
      new URL(c.req.url).pathname,
      c.res.status
    );
  });
});

// Helper for accessing the emitter anywhere in the call stack
export function getEmitter(): WideEventEmitter | undefined {
  return eventStorage.getStore();
}

Inside a repository or service layer, adding instrumentation is one line:

async function findUserById(id: string): Promise<User | null> {
  const start = Date.now();
  const user = await db.user.findUnique({ where: { id } });
  getEmitter()?.recordDbQuery(Date.now() - start);
  return user;
}

Cardinality Tradeoffs

The standard objection to wide events is cardinality. Storing user_id in a column store means potentially millions of distinct values. In Prometheus, this would be catastrophic. In ClickHouse, it is unremarkable.

Column stores handle high cardinality through late-bound aggregation. You store the raw data and aggregate at query time. The data is compressed efficiently (dictionary encoding for string columns, delta encoding for timestamps, LZ4 or ZSTD for everything). A ClickHouse table with one billion rows and two hundred columns is routinely queried in under a second for aggregation queries that touch five or six columns.

The tradeoff is storage cost versus query flexibility. Pre-aggregated metrics are cheap to store and fast to read because you threw away the detail at write time. Wide events retain every attribute, which costs more storage, but lets you answer questions you did not anticipate.

For most production workloads, wide events storage in ClickHouse or Parquet on S3 costs less than running a full metrics stack plus a log aggregation platform plus a tracing backend. The operational overhead of three separate systems compounds.

Tradeoffs Table

Dimension	Three-Pillar (Metrics + Logs + Traces)	Wide Events
Cardinality	Limited by metric label explosion	High cardinality is native
Correlation	Manual, requires shared IDs	Inherent, all fields on one record
Storage cost	Three separate systems, often overlapping	Single columnar store, well-compressed
Query flexibility	Constrained by pre-aggregation	Arbitrary slice and dice after the fact
Incident response	Three UIs, mental context switching	One query language, one UI
Sampling	Per-signal, inconsistent	Consistent per-request sampling
Operational overhead	High, three stacks to run	Lower, one ingestion pipeline
Real-time alerting	Native in metric systems	Requires streaming layer (Kafka, materialized views)
Historical data	Metrics retain aggregates; logs/traces often pruned	Full fidelity retained per event
Learning curve	Familiar, widely documented	Requires shift in mental model

The one area where the three-pillar model retains a real advantage is real-time alerting. Prometheus alerting rules evaluate against pre-aggregated data and can fire within seconds. Wide event alerting requires either a streaming aggregation layer (a materialized view in ClickHouse’s ReplicatedMergeTree or a Kafka Streams job) or accepting a slightly longer alert lag. For most operational alerts, a 30-60 second lag is acceptable. For strict SLA monitoring, you may want to keep a lightweight metric layer alongside your wide events.

Production Considerations

Sampling: Not every request needs a wide event stored permanently. Tail-based sampling works well here: sample 100% of requests that result in errors, timeouts, or high latency (p99+), and sample a smaller fraction of successful requests for baseline data. The emitter emits every event; the OTel collector or a sidecar decides what to forward to storage. This keeps storage costs bounded while retaining full fidelity for interesting requests.

PII: Wide events by nature contain user identifiers and request payloads. Establish a field-level PII policy before instrumentation scales. Scrub or hash fields like user_email and ip_address at the OTel collector layer, before data hits storage. Never rely on post-hoc deletion; columnar storage makes row-level deletion expensive.

Storage cost: ClickHouse with LZ4 compression routinely achieves 5-10x compression on wide event data. A table with 100 columns where most are sparse (null for many rows) compresses even better. Budget for raw data size divided by five as a starting point. Set a TTL on your ClickHouse table to expire data older than your retention policy automatically.

Query latency: Analytical queries on billions of rows can return in under a second if you partition correctly. Partition wide events by date. Cluster by service_name and http_path if those are your most frequent query dimensions. Avoid querying without a partition predicate in production dashboards.

Schema evolution: New fields should always be nullable. Adding a column to a ClickHouse table is instant and does not require backfilling. Removing a column requires more care. Treat your wide event schema as append-only for columns, and version breaking changes through a new table rather than schema mutation.

Migration Strategy

Moving from a three-pillar setup to wide events does not require a cutover. A parallel approach works better:

Start by emitting wide events alongside your existing instrumentation. Keep Prometheus, your log aggregation platform, and your tracing backend running. For one to two weeks, verify that your wide event data matches what you see in existing tools. Build a few dashboards against wide events for endpoints you know well.

Once you trust the data, start answering questions from wide events that you could not answer from your existing stack. “What is the p99 latency broken down by feature flag?” is a good first test. If you get a useful answer, the data model is working.

Deprecate pillar instrumentation endpoint by endpoint. Remove the Prometheus metrics for an endpoint after wide events cover the same alerting. Remove verbose log lines after wide events capture the same error context. Tracing spans for single-service calls can often be removed entirely; wide events capture the same timing information without the overhead of span propagation.

Distributed tracing retains value for cross-service requests where you need to understand which downstream service is responsible for latency. Keep spans for external service calls. Wide events at the entry point of each service, combined with trace ID correlation, gives you most of what a full trace provides at lower overhead.

Closing Thoughts

The three-pillar model is not wrong. It is the natural result of three technologies evolving independently and being bolted together. It works well enough that most teams never feel the friction until they are deep in an incident, jumping between four browser tabs, trying to manually correlate a trace ID with a log timestamp.

Wide events are a mental model shift as much as a technical one. Instead of asking “which pillar contains the answer to this question?” you ask “which attributes do I need to filter and aggregate?” The second question has a shorter path from question to answer, and it does not require you to have anticipated the question at instrumentation time.

The column store ecosystem, ClickHouse, DuckDB, Parquet on S3, has matured to the point where wide event analytics are operationally practical for teams without a dedicated platform engineering function. The instrumentation patterns using OpenTelemetry are standardized. The migration path is incremental, not a rewrite.

The cardinality limit you keep hitting in Prometheus is a design constraint of pre-aggregation, not an inherent property of observability. Wide events remove that constraint.