Engineering Metrics That Matter: DORA, Developer Experience, and What Not to Measure

Every engineering org eventually starts measuring things. The problem is rarely a lack of data. It’s measuring things that feel productive while optimizing for the wrong outcomes.

Story points velocity goes up every sprint. Lead time quietly creeps from two days to two weeks. Nobody notices until a competitor ships what you planned six months ago.

This guide covers what actually matters: the DORA four key metrics with real collection methods, developer experience signals as leading indicators, and the specific vanity metrics that actively harm teams. There’s also a Goodhart’s Law section because every team eventually hits this, and a lightweight implementation you can run from GitHub Actions today.

The DORA Four Key Metrics

The DevOps Research and Assessment (DORA) program identified four metrics that consistently distinguish high-performing engineering teams from low performers. The research covers thousands of organizations over nearly a decade. These four predict software delivery performance and organizational outcomes better than anything else studied at that scale.

Deployment Frequency: How often you successfully deploy to production.

Lead Time for Changes: The time from commit to production.

Change Failure Rate: The percentage of deployments that cause a production incident requiring remediation.

Time to Restore Service: How long it takes to recover from a production failure.

Here are the 2023 benchmarks to orient where your team sits:

Metric	Elite	High	Medium	Low
Deployment Frequency	Multiple times/day	1x/day to 1x/week	1x/week to 1x/month	Fewer than 1x/month
Lead Time for Changes	< 1 hour	1 day to 1 week	1 week to 1 month	> 6 months
Change Failure Rate	0-5%	5-10%	10-15%	> 15%
Time to Restore Service	< 1 hour	< 1 day	1 day to 1 week	> 1 week

If your team deploys once a week and takes four hours to recover from an incident, you’re in the medium tier. That’s a starting point, not a verdict.

Collecting DORA Metrics Without Expensive Tooling

You don’t need Accelerate or LinearB or Faros to collect these. GitHub’s API and a deployment tracking table get you 80% of the way.

Here’s a TypeScript script that pulls deployment frequency and lead time from GitHub:

import { Octokit } from "@octokit/rest";

interface DeploymentRecord {
  deployedAt: Date;
  commitSha: string;
  environment: string;
}

interface LeadTimeResult {
  deploymentSha: string;
  deployedAt: Date;
  firstCommitAt: Date;
  leadTimeHours: number;
}

async function collectLeadTime(
  owner: string,
  repo: string,
  since: Date
): Promise<LeadTimeResult[]> {
  const octokit = new Octokit({ auth: process.env.GITHUB_TOKEN });

  // Pull deployments from the GitHub Deployments API
  const { data: deployments } = await octokit.repos.listDeployments({
    owner,
    repo,
    environment: "production",
    per_page: 100,
  });

  const results: LeadTimeResult[] = [];

  for (const deployment of deployments) {
    const deployedAt = new Date(deployment.created_at);
    if (deployedAt < since) continue;

    // Find the previous production deployment to bound the commit window
    const deploymentIndex = deployments.indexOf(deployment);
    const previousDeployment = deployments[deploymentIndex + 1];
    const previousSha = previousDeployment?.sha ?? "";

    // List commits between previous deploy and this one
    let commits: { commit: { author: { date?: string } } }[] = [];

    if (previousSha) {
      const { data: comparison } = await octokit.repos.compareCommits({
        owner,
        repo,
        base: previousSha,
        head: deployment.sha,
      });
      commits = comparison.commits;
    }

    if (commits.length === 0) continue;

    // Lead time = time from the oldest commit in this batch to deploy
    const commitDates = commits
      .map((c) => new Date(c.commit.author?.date ?? ""))
      .filter((d) => !isNaN(d.getTime()))
      .sort((a, b) => a.getTime() - b.getTime());

    const firstCommitAt = commitDates[0];
    const leadTimeHours =
      (deployedAt.getTime() - firstCommitAt.getTime()) / (1000 * 60 * 60);

    results.push({
      deploymentSha: deployment.sha,
      deployedAt,
      firstCommitAt,
      leadTimeHours,
    });
  }

  return results;
}

async function deploymentFrequency(
  owner: string,
  repo: string,
  windowDays: number
): Promise<number> {
  const octokit = new Octokit({ auth: process.env.GITHUB_TOKEN });
  const since = new Date(Date.now() - windowDays * 24 * 60 * 60 * 1000);

  const { data: deployments } = await octokit.repos.listDeployments({
    owner,
    repo,
    environment: "production",
    per_page: 100,
  });

  const deploymentsInWindow = deployments.filter(
    (d) => new Date(d.created_at) >= since
  );

  return deploymentsInWindow.length / windowDays; // deployments per day
}

For change failure rate and time to restore, you need incident data. The simplest approach: tag GitHub issues with a incident label and a production label when something breaks in production, then close them when resolved.

interface IncidentRecord {
  openedAt: Date;
  closedAt: Date | null;
  linkedDeploymentSha: string | null;
}

async function collectIncidents(
  owner: string,
  repo: string,
  since: Date
): Promise<IncidentRecord[]> {
  const octokit = new Octokit({ auth: process.env.GITHUB_TOKEN });

  const { data: issues } = await octokit.issues.listForRepo({
    owner,
    repo,
    labels: "incident,production",
    state: "all",
    since: since.toISOString(),
    per_page: 100,
  });

  return issues.map((issue) => ({
    openedAt: new Date(issue.created_at),
    closedAt: issue.closed_at ? new Date(issue.closed_at) : null,
    // Convention: include the deploy SHA in the issue body as "deploy: abc1234"
    linkedDeploymentSha: extractDeploySha(issue.body ?? ""),
  }));
}

function extractDeploySha(body: string): string | null {
  const match = body.match(/deploy:\s*([a-f0-9]{7,40})/i);
  return match?.[1] ?? null;
}

function timeToRestoreHours(incidents: IncidentRecord[]): number {
  const resolved = incidents.filter((i) => i.closedAt !== null);
  if (resolved.length === 0) return 0;

  const totalHours = resolved.reduce((sum, i) => {
    return sum + (i.closedAt!.getTime() - i.openedAt.getTime()) / (1000 * 60 * 60);
  }, 0);

  return totalHours / resolved.length;
}

This runs in a GitHub Actions workflow on a schedule, writes results to a JSON file in the repo (or a simple SQLite database for longer history), and gives you a 30-day rolling view of all four metrics.

Developer Experience Metrics: The Leading Indicators DORA Misses

DORA metrics are lagging indicators. Deployment frequency tells you what happened. Developer experience (DX) metrics tell you what’s about to happen.

Four DX signals that correlate strongly with future DORA performance:

Build Times: A CI pipeline that takes 22 minutes trains engineers to batch work and avoid running it locally. Measure p50 and p95 build times per week. If p95 exceeds 15 minutes, the team is paying a compounding tax on every PR.

PR Review Latency: Time from PR opened to first substantive review (not just a “LGTM” rubber stamp). Over 24 hours in a co-located team is a process problem. Over 48 hours in an async team needs investigation. This is often the single largest contributor to lead time.

Environment Setup Time: How long does it take a new engineer to run the app locally from a clean checkout? If the honest answer is “a day or two,” that’s not an onboarding problem. It’s a system complexity problem that affects every engineer every time they reset their environment.

Cognitive Load Surveys: Quarterly, ask engineers three questions on a 1-5 scale: “I understand what I need to do to ship safely,” “I can find what I need without asking someone,” “The tools help more than they hinder.” Track the trend, not the absolute score.

These are collected differently from DORA metrics. Build times come from your CI system’s API. PR review latency comes from GitHub. Environment setup time and cognitive load come from periodic team surveys. Use a simple Google Form or Typeform. A spreadsheet is enough for a team under 30.

Vanity Metrics to Avoid

These metrics feel productive to track and create real damage when acted on.

Lines of Code: Measures nothing useful. A well-refactored module that deletes 400 lines and adds 50 is better than the reverse. Teams that track this ship bloated, unmaintainable code.

Story Points Velocity: Useful for sprint planning in a stable team. Harmful when used to compare teams, measure productivity, or set expectations with stakeholders. Points are calibrated per team, per sprint. Velocity increases when teams game the estimation, not when they ship more.

Commit Counts: Conflates activity with progress. Engineers who commit frequently (small, clean commits) look more productive than engineers who commit once at the end of a careful, considered implementation. This trains the wrong behavior.

Test Coverage Percentage: Coverage at 80% means 80% of lines are executed by tests. It says nothing about what’s being asserted, whether the tests catch real bugs, or whether the covered code is the code that fails in production. Teams that chase 90% coverage write assertions that always pass.

The pattern with all of these: they’re easy to measure and easy to game, which makes them worse than useless once people know they’re being tracked.

How Metrics Change by Team Size

A 3-person team and a 30-person team need different metrics, different collection methods, and different interpretations.

3-person team: Deployment frequency and lead time are the only DORA metrics worth tracking weekly. Change failure rate and time to restore are better tracked as incidents per quarter. At this size, cognitive load surveys are a conversation over lunch, not a form. PR review latency matters, but everyone knows who’s slow without a dashboard.

10-person team: All four DORA metrics become meaningful. PR review latency starts showing systemic patterns (certain areas of the codebase, certain team members as bottlenecks). Build time starts mattering at this scale because 10 engineers each running CI multiple times a day accumulates fast.

30-person team: You now have enough signal to segment metrics by team or squad. Deployment frequency per squad, lead time per squad. Cross-squad PR review latency reveals coordination overhead. Environment setup time differences between squads surface infrastructure inconsistencies. At this size, a lightweight dashboard becomes worth the investment. At three people, a spreadsheet updated weekly is enough.

The Goodhart’s Law Trap

“When a measure becomes a target, it ceases to be a good measure.”

This is the most predictable failure mode in engineering metrics. It’s not theoretical. It happens at almost every organization that starts measuring.

Deployment frequency becomes a target: engineers start merging feature flags to technically deploy to production, then shipping features off by default. Deployment count goes up, actual feature delivery stays flat.

Lead time becomes a target: teams start breaking features into smaller PRs specifically to reduce measured lead time, losing coherent scope and creating integration risk.

Change failure rate becomes a target: incidents get reclassified. “That wasn’t really a production failure, it was a configuration issue.” The metric improves, reliability does not.

The mitigation is not to avoid targets. It’s to measure multiple signals simultaneously so gaming one metric visibly degrades another. If deployment frequency goes up but lead time and change failure rate hold steady, you’re probably improving. If deployment frequency goes up and change failure rate spikes, you’ve found a problem.

The other mitigation: keep the conversation about outcomes (can users rely on the product?) rather than metrics (did we hit the deployment frequency target?).

A Lightweight Metrics Dashboard with GitHub Actions

Here’s a GitHub Actions workflow that runs the collection scripts above on a schedule and writes results to a metrics file in your repo:

name: Engineering Metrics Collection

on:
  schedule:
    - cron: "0 9 * * 1" # Every Monday at 9am UTC
  workflow_dispatch:

jobs:
  collect-metrics:
    runs-on: ubuntu-latest
    permissions:
      contents: write
      issues: read
      deployments: read

    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-node@v4
        with:
          node-version: "20"

      - name: Install dependencies
        run: npm ci

      - name: Collect DORA metrics
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          REPO_OWNER: ${{ github.repository_owner }}
          REPO_NAME: ${{ github.event.repository.name }}
        run: npx ts-node scripts/collect-metrics.ts

      - name: Commit metrics update
        run: |
          git config user.name "metrics-bot"
          git config user.email "metrics-bot@noreply"
          git add metrics/
          git diff --staged --quiet || git commit -m "chore: weekly metrics update $(date -u +%Y-%m-%d)"
          git push

The TypeScript entry point:

import * as fs from "fs";
import * as path from "path";
import { collectLeadTime, deploymentFrequency, collectIncidents, timeToRestoreHours } from "./metrics";

async function main() {
  const owner = process.env.REPO_OWNER!;
  const repo = process.env.REPO_NAME!;
  const since = new Date(Date.now() - 30 * 24 * 60 * 60 * 1000); // 30 days

  const [freq, leadTimes, incidents] = await Promise.all([
    deploymentFrequency(owner, repo, 30),
    collectLeadTime(owner, repo, since),
    collectIncidents(owner, repo, since),
  ]);

  const avgLeadTimeHours =
    leadTimes.length > 0
      ? leadTimes.reduce((s, r) => s + r.leadTimeHours, 0) / leadTimes.length
      : 0;

  const mttRHours = timeToRestoreHours(incidents);

  const result = {
    collectedAt: new Date().toISOString(),
    windowDays: 30,
    deploymentFrequencyPerDay: freq,
    avgLeadTimeHours,
    incidentCount: incidents.length,
    avgTimeToRestoreHours: mttRHours,
  };

  const outPath = path.join("metrics", `${result.collectedAt.slice(0, 10)}.json`);
  fs.mkdirSync("metrics", { recursive: true });
  fs.writeFileSync(outPath, JSON.stringify(result, null, 2));

  console.log("Metrics collected:", result);
}

main().catch(console.error);

This is intentionally simple. The metrics directory becomes a time-series of JSON files. You can plot them in a GitHub Actions summary, a Notion embed, or a simple static HTML page. The important thing is that collection happens automatically and the data persists somewhere outside anyone’s memory.

What a Fractional CTO Looks at in the First Week

When coming into an unfamiliar engineering org, the first week is mostly listening and reading. But a few specific signals surface quickly.

Lead time for changes, measured informally by asking “when did you start this feature?” and “when did it ship?” If those two dates are weeks apart for features that shouldn’t take weeks, the constraint is somewhere in the process, not the engineers.

PR age: open your GitHub pull requests tab and sort by oldest. PRs open for more than a week are a coordination problem, a review culture problem, or a scope problem. Reading the oldest three tells you which.

Deployment process: can anyone deploy to production, or does it go through one person? Single-point-of-failure deployment processes are both a reliability risk and a lead time killer.

Incident history: the last five production incidents. Not the postmortems (those are often sanitized). The raw Slack threads or PagerDuty alerts. How long did it take to detect? How long to resolve? Did the same class of failure occur more than once?

These are all observable in the first week without any tooling. Tooling comes later, once you understand what you’re actually measuring and why.

Closing

The goal of engineering metrics is not to have a dashboard. It’s to make slow problems visible before they become urgent ones. DORA gives you the lagging indicators. Developer experience signals give you the leading ones. Vanity metrics give you the illusion of progress while the real problems compound.

Start with lead time and deployment frequency. Add incident tracking when you have enough deployments to make the percentages meaningful. Run a cognitive load survey before you invest in tooling, because sometimes the answer is simpler than you expect.

Measure what you can act on. Everything else is noise.