Observability for Small Teams: What to Monitor When You Can't Afford a Platform Team
A practical guide to setting up logs, metrics, and traces for teams of 1-5 engineers. Covers what actually matters at small scale, tool recommendations with free tiers, SLO-based alerting that avoids alert fatigue, and a phased rollout plan.
Most observability guides are written by platform teams for platform teams. They assume you have a dedicated SRE rotation, a Datadog contract with a six-figure line item, and engineering hours to burn on configuring dashboards nobody reads.
If you are a team of two to five engineers shipping a product, that context does not apply. Your constraints are different: no dedicated infra engineer, a tight budget, and incidents where you are the on-call engineer, the incident commander, and the person writing the post-mortem at the same time.
This guide is about what actually matters at that scale, what you should skip entirely, and how to build a setup that stays useful as you grow.
The Three Pillars, Reframed for Small Teams
The three-pillar model (logs, metrics, traces) is real. But the weight you assign each pillar should match your debugging workflow, not a textbook.
For most small teams:
- Logs answer “what happened” and are the first thing you reach for during an incident.
- Metrics answer “is something wrong right now” and drive your alerts.
- Traces answer “why is this slow” and matter once you have multiple services or a meaningful latency problem.
The common mistake is treating all three equally and setting them up in parallel from the start. A four-engineer product team does not need distributed tracing on day one. Start with structured logs and a handful of metrics. Add traces when you have a latency problem that logs alone cannot explain.
What NOT to Monitor
Before covering what to track, it is worth being explicit about what to skip, because this is where most small teams waste their time.
Skip:
- CPU and memory utilization as primary alert signals. These are trailing indicators. By the time CPU spikes, the impact has already happened. Alert on the symptom (error rate, latency) not the infrastructure metric.
- Per-request trace spans for simple CRUD APIs. The overhead is real and the insight is marginal if you are running a monolith on a single service.
- Log ingestion for every request. Access logs at full volume on Grafana Cloud or Axiom will eat your free tier within days. Filter at the source.
- More than five or six dashboards. If you have fifteen panels that nobody opens, you do not have observability. You have decoration.
- P50 latency as an alert target. P50 is almost always fine. Alert on P95 or P99. The worst 5% of your users are the first to churn.
Tool Choices for the Budget
Here are the tools worth considering at small scale, with honest assessments.
| Tool | What it is | Free tier | Cost at scale |
|---|---|---|---|
| Grafana Cloud | Metrics + logs + traces | 50GB logs/mo, 10K series | ~$8/mo per 100GB logs |
| Axiom | Log analytics | 500GB/mo free | $25/mo flat for most startups |
| Betterstack (Logtail) | Logs + uptime | 1GB/day | $25/mo for 3GB/day |
| OpenTelemetry Collector | Telemetry pipeline | Free (self-hosted) | Infrastructure cost only |
| Grafana Alloy | OTEL-based agent | Free | Infrastructure cost only |
| Sentry | Error tracking | 5K errors/mo | $29/mo |
| UptimeRobot | Uptime checks | 50 monitors | $7/mo |
For a team under five engineers, the combination that covers most needs with zero cost: Grafana Cloud free tier for metrics and alerts, Axiom or Betterstack for logs, Sentry for error tracking. You can run this stack for months without spending anything, which matters when you are still finding product-market fit.
The important framing: these tools are not permanent decisions. OpenTelemetry as your instrumentation layer means you can swap backends without re-instrumenting your application. Instrument once, route anywhere.
Structured Logging: The Foundation
The most common logging mistake is writing unstructured strings. When your logs are sentences, you cannot filter, aggregate, or alert on them reliably.
// Bad: string concatenation you cannot query
logger.info(`User ${userId} created order ${orderId} for $${amount}`);
// Good: structured JSON you can filter and aggregate
logger.info({
event: "order.created",
userId,
orderId,
amountCents,
durationMs: Date.now() - startTime,
});
Set up a logger wrapper early so every log line is structured consistently. Here is a minimal one that works with Axiom, Grafana Cloud Loki, or Betterstack:
import pino from "pino";
const isDev = process.env.NODE_ENV !== "production";
export const logger = pino({
level: process.env.LOG_LEVEL ?? "info",
...(isDev
? { transport: { target: "pino-pretty" } }
: {
formatters: {
level: (label) => ({ level: label }),
},
base: {
service: process.env.SERVICE_NAME ?? "app",
env: process.env.NODE_ENV,
version: process.env.APP_VERSION,
},
}),
});
Three fields every log line should carry: event (what happened, dot-notation), durationMs (how long it took), and a correlation ID you can trace through a request. The correlation ID is the single cheapest investment in debuggability:
import { AsyncLocalStorage } from "async_hooks";
interface RequestContext {
requestId: string;
userId?: string;
}
const requestContext = new AsyncLocalStorage<RequestContext>();
export function withRequestContext<T>(
ctx: RequestContext,
fn: () => T
): T {
return requestContext.run(ctx, fn);
}
export function getRequestContext(): RequestContext | undefined {
return requestContext.getStore();
}
// In your middleware
app.use((req, res, next) => {
withRequestContext(
{
requestId: req.headers["x-request-id"] as string ?? crypto.randomUUID(),
userId: req.user?.id,
},
next
);
});
// In your logger wrapper
export function log(level: "info" | "warn" | "error", fields: Record<string, unknown>) {
const ctx = getRequestContext();
logger[level]({ ...ctx, ...fields });
}
Now every log line from within a request handler automatically carries the request ID. When something goes wrong, you grep for the request ID and see the full request lifecycle in order.
Metrics: Five That Matter, Not Fifty
Metrics are where most teams over-invest early. The goal is not comprehensive coverage. The goal is knowing within two minutes whether something is wrong with your product.
The five metrics worth instrumenting from the start:
- HTTP error rate (5xx responses as a percentage of total requests)
- HTTP P95 latency per route, or at minimum per service
- Job queue depth if you have background jobs
- Database query P95 latency (connection pool exhaustion shows up here first)
- Business event rate (orders per minute, signups per hour, whatever your core action is)
The business event rate is the most underrated metric. A drop in order rate at 2 PM on a Tuesday is a clearer signal than any infrastructure metric. It tells you your product is broken from a user’s perspective, not from a server’s perspective.
Here is a minimal Prometheus-compatible metrics setup using the prom-client library:
import { Counter, Histogram, Registry, collectDefaultMetrics } from "prom-client";
export const registry = new Registry();
collectDefaultMetrics({ register: registry });
export const httpRequestDuration = new Histogram({
name: "http_request_duration_ms",
help: "HTTP request duration in milliseconds",
labelNames: ["method", "route", "status_code"],
buckets: [10, 50, 100, 200, 500, 1000, 2000, 5000],
registers: [registry],
});
export const httpRequestErrors = new Counter({
name: "http_request_errors_total",
help: "Total number of HTTP errors",
labelNames: ["method", "route", "status_code"],
registers: [registry],
});
export const businessEvents = new Counter({
name: "business_events_total",
help: "Core business events",
labelNames: ["event_type", "outcome"],
registers: [registry],
});
// Express middleware
export function metricsMiddleware(
req: express.Request,
res: express.Response,
next: express.NextFunction
) {
const start = Date.now();
res.on("finish", () => {
const duration = Date.now() - start;
const route = req.route?.path ?? "unknown";
const labels = { method: req.method, route, status_code: String(res.statusCode) };
httpRequestDuration.observe(labels, duration);
if (res.statusCode >= 500) {
httpRequestErrors.inc(labels);
}
});
next();
}
// Expose metrics endpoint
app.get("/metrics", async (req, res) => {
res.set("Content-Type", registry.contentType);
res.end(await registry.metrics());
});
Grafana Cloud’s free tier scrapes this endpoint via a Prometheus-compatible remote write. Set the scrape interval to 30 seconds for a small team. One-minute resolution is fine for incidents you will respond to in minutes, not milliseconds.
Alerting Without Alert Fatigue
Alert fatigue is the failure mode that makes the entire observability investment worthless. When alerts fire constantly, engineers learn to ignore them. When a real incident hits, it looks the same as the noise.
The root cause is almost always the same: too many alerts, thresholds set on average values, and no quiet period between repeated fires.
Rules that actually reduce alert fatigue:
Alert on rate, not on count. An error count of 50 might be normal for a high-traffic endpoint and catastrophic for a low-traffic one. Error rate (errors / total requests) normalizes for traffic.
Use multi-window SLO-based alerting instead of static thresholds. A five-minute spike might be a blip. The same elevated error rate sustained for thirty minutes is an incident.
Set a minimum volume filter. If your endpoint gets two requests per hour, a single error produces a 50% error rate. Add a condition: only alert when request volume in the window exceeds N.
// Not an alerting DSL, but the logic your alert rule should express:
interface AlertCondition {
// Error rate over a short window (fast detection)
shortWindow: {
durationMinutes: 5;
errorRateThreshold: 0.05; // 5%
minRequestVolume: 20;
};
// Error rate over a longer window (confirmation it's not a blip)
longWindow: {
durationMinutes: 60;
errorRateThreshold: 0.02; // 2%
minRequestVolume: 100;
};
// Alert fires only when BOTH windows are breaching
combinator: "AND";
}
This two-window pattern (short window for detection, long window for confirmation) is the simplest approximation of proper SLO-based alerting and eliminates most false positives.
Define a maximum of three alert severity levels and be strict about which is which:
- Page immediately: user-facing functionality is broken right now. Error rate over 5%, P95 latency over 5 seconds, payment processing failures.
- Notify during business hours: degraded but not broken. Error rate elevated, latency creeping up, job queue growing.
- Log for review: anomalies that need investigation but are not causing user harm right now.
Page-worthy alerts should be rare. If you are getting paged more than once a week on the same system, the alert threshold is wrong or the underlying problem is unfixed.
SLO-Based Monitoring: The Minimum Viable Version
Service Level Objectives are not enterprise overhead. They are the clearest way to answer “is our product working well enough?” without guessing at thresholds.
A minimal SLO for a startup: pick one availability target and one latency target, define what “good” means for each, and measure the burn rate.
interface SLO {
name: string;
// What percentage of requests must succeed
availabilityTarget: number; // e.g. 0.99 = 99%
// What latency must what percentage of requests be under
latencyTarget: {
percentile: number; // e.g. 0.95
thresholdMs: number; // e.g. 1000
};
// Rolling window
windowDays: 30;
}
const orderServiceSLO: SLO = {
name: "order-service",
availabilityTarget: 0.99,
latencyTarget: {
percentile: 0.95,
thresholdMs: 800,
},
windowDays: 30,
};
The error budget is what makes SLOs actionable: if your SLO is 99% availability over 30 days, you have 0.01 * 30 * 24 * 60 = 432 minutes of allowable downtime per month. When that budget is 50% consumed in the first week, you are on track to miss your SLO. That is when you stop shipping features and fix reliability.
In Grafana, you can approximate this with two recording rules and a single alert:
- Recording rule 1: availability over 30 days (total 5xx / total requests)
- Recording rule 2: current 1-hour burn rate (error rate now / monthly error budget rate)
- Alert: fires when the 1-hour burn rate exceeds 14x (meaning you will burn your entire monthly budget in 2 days at this rate)
The 14x threshold is not magic. It comes from a specific SLO consumption pace, and most observability tools expose it as a “fast burn” alert preset.
Distributed Tracing: When to Add It
Add tracing when you have a latency problem and you cannot explain it from logs and metrics alone. That is the right signal. Not “we have more than one service”, not “we are a real company now.”
If you do add tracing, use OpenTelemetry from the start so the backend is swappable:
import { NodeSDK } from "@opentelemetry/sdk-node";
import { getNodeAutoInstrumentations } from "@opentelemetry/auto-instrumentations-node";
import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-http";
const sdk = new NodeSDK({
traceExporter: new OTLPTraceExporter({
url: process.env.OTEL_EXPORTER_OTLP_ENDPOINT,
headers: {
authorization: `Bearer ${process.env.OTEL_API_KEY}`,
},
}),
instrumentations: [
getNodeAutoInstrumentations({
// Disable fs instrumentation — it generates enormous volume for no value
"@opentelemetry/instrumentation-fs": { enabled: false },
}),
],
});
sdk.start();
Auto-instrumentation covers HTTP clients, database queries (pg, mysql2, mongoose), and framework middleware automatically. You get useful traces without manual span creation for most production incidents.
The places where manual span creation pays off: external API calls with custom retry logic, business-critical transaction boundaries (checkout flow, billing webhook processing), and background jobs where you want to attribute latency to specific steps.
A Phased Rollout Plan
Trying to implement all of this in one sprint creates the same problem as building fifteen dashboards: you end up with a half-finished setup you never fully trust.
Phase 1 (Day 1): Basic visibility
- Structured logging with correlation IDs
- Sentry for error tracking
- Uptime check on your main endpoint (UptimeRobot or Betterstack)
- One alert: if uptime check fails for 5 minutes, page the on-call engineer
This alone is better than 80% of early-stage startups.
Phase 2 (Week 2): Meaningful metrics
- Add prom-client and instrument HTTP error rate + P95 latency
- Ship metrics to Grafana Cloud free tier
- Two alerts: error rate over 5% (5-minute window), P95 over 3 seconds (5-minute window)
- One dashboard: error rate and latency for your main service
Phase 3 (Month 2): SLO and business metrics
- Define one SLO per user-facing service
- Add business event metrics (orders, signups, payments)
- Set up the fast-burn SLO alert
- Start tracking error budget weekly
Phase 4 (When you have a latency incident): Add OpenTelemetry tracing.
Do not do Phase 4 before you need it. The instrumentation is low overhead, but the cognitive overhead of reading traces when you have no baseline is high.
What Good Looks Like at Five Engineers
A small team with healthy observability can answer these questions within two minutes of hearing about an incident:
- Is this still happening now?
- When did it start?
- Which endpoints or jobs are affected?
- Is the error rate or latency elevated?
- What does the relevant log output show?
If you can answer those five questions, you have enough observability for your scale. The goal is not comprehensive coverage. It is fast time-to-understand during an incident.
Dashboards nobody reads, alerts that fire constantly, and traces on every request are not observability. They are noise. The discipline is knowing what to leave out.
Tradeoffs: Managed vs Self-Hosted
| Approach | Cost | Maintenance | Data residency | When to use |
|---|---|---|---|---|
| Grafana Cloud (free) | Free up to limits | None | US/EU options | Default for most startups |
| Axiom | Free 500GB/mo | None | US | High log volume on free tier |
| Betterstack | Free 1GB/day | None | EU available | GDPR-sensitive workloads |
| Self-hosted Grafana + Loki | Infrastructure cost | High | You control it | Compliance requirements, >1TB/day |
| Datadog | Expensive | Low | Limited | After Series A, dedicated platform team |
The managed options cover most teams until well past Series A. Self-hosted only makes sense when compliance forces it or when volume makes the per-GB cost of managed tools larger than the cost of running your own infrastructure plus the engineering time to maintain it.
For most five-person teams, the answer is Grafana Cloud free tier for metrics, Axiom or Betterstack for logs, and Sentry for errors. That stack costs nothing, takes one sprint to set up properly, and gives you everything you need to run on-call effectively.
Observability at this scale is not about comprehensive coverage. It is about having enough signal to understand incidents quickly and enough discipline to not drown in noise between them.
More in DevOps
How Terraform Works Internally: HCL Parsing, Dependency Graph Execution, and the Provider Protocol Behind Infrastructure as Code
A deep dive into Terraform's internal architecture covering HCL parsing, the expression evaluation engine, DAG-based dependency resolution, the gRPC provider plugin protocol, state mechanics, and the plan/apply lifecycle.
How Docker Works Internally: Namespaces, Cgroups, Union Filesystems, and the OCI Runtime Behind Every Container
A deep dive into what happens when you run docker run: Linux namespaces for process isolation, cgroups v2 for resource enforcement, overlay2 union filesystem for layered images, the OCI runtime spec and how runc spawns containers, content-addressable storage, the containerd shim architecture, and networking with veth pairs.
How Kubernetes Works: Pods, Scheduling, and the Reconciliation Loop
A deep dive into Kubernetes internals covering the control plane architecture, etcd watch mechanics, the scheduler's filter and score phases, controller manager reconciliation loops, kubelet pod assignment, the CRI and CNI plugin models, and production considerations for resource requests, pod disruption budgets, and cluster autoscaler behavior.
How Docker Works: Namespaces, Cgroups, Union Filesystems, and the Container Runtime from Build to Execution
A deep dive into Docker internals covering Linux namespace isolation, cgroup resource constraints, OverlayFS copy-on-write layer mechanics, Dockerfile build cache invalidation rules, the containerd and runc OCI runtime stack, and production considerations for image hardening and startup latency.