23. Observability — Knowing Before Users Tell You
“The worst monitoring system is your users. They’re slow, they’re unreliable, and they tell you on Twitter.”
The story: the dashboard in the cockpit
Section titled “The story: the dashboard in the cockpit”A pilot doesn’t fly by looking out of the window and waiting to see if the ground gets closer. The cockpit is full of instruments: altitude, airspeed, fuel, engine temperature, each with a normal range. Some are watched constantly; others only matter when an alarm sounds. The alarms are carefully designed: few, loud, specific, and each with a checklist. An alarm that went off every five minutes for no reason would be taped over within a week — and then, one day, it would be right.
Your production website is an aircraft full of passengers you can’t see. Without instruments, you learn about problems the way most teams do: a customer support ticket, a tweet, the CEO’s nephew. By then, the problem has been hurting people for hours or days.
Observability is the cockpit: the data your system emits (errors, metrics, logs, traces), the instruments that show it (dashboards), and the alarms (alerts) that wake the right person at the right time — and not otherwise.
The four signals for a frontend
Section titled “The four signals for a frontend”For backend services, the classic advice from Google’s Site Reliability Engineering practice is to watch the four golden signals: latency, traffic, errors and saturation. Translated for a frontend product:
- Latency → Core Web Vitals and other timings from real users (RUM): LCP, INP, CLS, TTFB, API latency as experienced in the browser.
- Traffic → page views, sessions, key flow starts (search, checkout), API request volume.
- Errors → JavaScript exceptions, unhandled promise rejections, failed network requests, failed resource loads (a CDN outage often shows up first as a spike in failed script loads), server 5xx rates.
- Saturation → server CPU/memory/disk on your VPS, database connections, and on the client side, long-task frequency and memory pressure.
Plus one frontend-specific concern: correctness of what users see — is the page actually rendering the right content? (Synthetic checks catch “returns 200 but is blank.”)
Error monitoring
Section titled “Error monitoring”When a JavaScript error happens in Priya’s browser, it happens nowhere you can see — unless you capture it. Error monitoring tools (Sentry is the best known; there are self-hostable alternatives such as GlitchTip, and many observability platforms include it) install a small SDK that:
- hooks
window.onerrorandunhandledrejection(and framework error boundaries); - captures the stack trace, the browser and OS, the URL, the release version, breadcrumbs (the clicks, navigations, console messages and network requests leading up to the error) and optional user context;
- sends it to a backend that groups identical errors into issues, counts occurrences and affected users, and alerts on new or spiking issues.
What makes error monitoring useful rather than noisy:
- Release tagging: every event carries the deploy version, so you can see “this error started in release 2026.11.20-1430” and roll back (Chapter 4).
- Grouping and triage: thousands of events, a dozen issues, ranked by users affected.
- Filtering noise: browser extensions, ad blockers, bots and ancient browsers throw errors that aren’t yours. Filter by script origin (only errors from your code), ignore known third-party noise, sample high-volume issues.
- Ownership: issues routed to the team that owns the code.
Source maps in production
Section titled “Source maps in production”Production code is minified (Chapter 1): a stack trace says a.js:1:48213, which is useless. The fix is source maps — but not public ones. The secure pattern:
- Your build generates source maps (with hidden mode, so bundles don’t reference them).
- CI uploads the maps to your error tracker, tagged with the release (modern tooling links bundles and maps with debug IDs embedded in both, avoiding filename-matching problems).
- CI does not deploy the maps to the public web server (or deletes them after upload).
- The error tracker symbolicates incoming stack traces privately:
Essay.tsx:42 in LikeButton.
You get readable stack traces; the public gets no source code.
Session replay — handle with care
Section titled “Session replay — handle with care”Many tools offer session replay: recording DOM changes and interactions so you can watch a reconstruction of what the user did before an error. It’s extraordinarily useful for debugging — and a serious privacy matter. Good tools mask all text and inputs and block media by default; keep those defaults, unmask only specific, reviewed elements, and never record pages with sensitive data (payments, health, messages). Sample sparingly.
RUM vs. synthetic monitoring
Section titled “RUM vs. synthetic monitoring”Two complementary ways to watch performance and availability:
- RUM (real-user monitoring, Chapter 17): measures what real users experience. Truthful, broad, segmented by device and network. But: it needs traffic (no users at 3 a.m. means no data at 3 a.m.), it’s noisy, and it tells you about problems after users hit them.
- Synthetic monitoring: scripted checks from fixed locations on a schedule — load the homepage every 5 minutes from Mumbai, Singapore and Frankfurt; run the login→search→save flow every 15 minutes. Consistent, works with zero traffic, catches outages and regressions immediately, and gives clean baselines. But: it’s not your users — one device, one network, a fresh cache.
Use both: synthetic for “is it up and is it the same as yesterday,” RUM for “how is it for real people.”
Logs are timestamped records of events: requests handled, errors thrown, jobs run. For a VPS-hosted app, you’ll have nginx access and error logs, your Node app’s logs, and system logs. Make them useful:
- Structured: JSON lines with consistent fields (
time,level,msg,route,status,duration_ms,release,request_id) instead of free text. Searchable and aggregatable. - Correlated: a request ID generated at nginx (or the app) and passed through every log line for that request — and returned in a response header so a frontend error report can reference the backend logs for the same request. (Distributed tracing, e.g. with OpenTelemetry, generalizes this into timing trees across services.)
- Leveled and sampled: debug logs off in production; high-volume success logs sampled.
- Retained deliberately: logs contain personal data (IPs, user IDs, URLs); keep them only as long as you need.
- Centralized: shipped off the box (so you can still read them when the box is the problem) to a log store you can query.
Uptime checks
Section titled “Uptime checks”The simplest, most valuable monitor: an external service requests your site every minute from several regions and alerts if it fails. Check more than “returns 200”:
- the homepage returns 200 and contains expected text (a blank page served with 200 is a classic outage — the CDN served an error page, or the app shell rendered with no data);
- an API health endpoint returns 200 and checks its dependencies (database reachable);
- the TLS certificate isn’t about to expire (renewal automation fails silently more often than you’d think);
- DNS resolves correctly;
- (Chapter 22) the homepage doesn’t contain
noindex.
Alerting — and alert fatigue
Section titled “Alerting — and alert fatigue”An alert interrupts a human. That’s expensive, so every alert must be:
- Actionable — the person receiving it can and should do something now.
- About symptoms users feel — “checkout error rate > 2% for 5 minutes,” “p75 INP on mobile > 500 ms since the last release,” “homepage down from 2 of 3 regions” — rather than causes that may not matter (“CPU at 85%”). Cause-based signals go on dashboards, not pagers.
- Urgent — if it can wait until morning, it’s a ticket, not a page.
- Owned — routed to a specific on-call person or team, with a runbook link.
Alert fatigue is what happens when you don’t follow these rules: dozens of alerts a day, most meaningless; people mute channels; and the real one gets missed. It’s the most common failure mode of monitoring. The cure is ruthless: every alert that fired without requiring action gets fixed (threshold changed, converted to a ticket, or deleted) that same week.
Good alert design uses thresholds over a window (5 minutes of elevated errors, not one spike), relative changes after deploys (error rate doubled vs the previous release), and multi-region confirmation for uptime (two of three locations failing).
Dashboards
Section titled “Dashboards”A good dashboard answers one question at a glance. Make a few, each for an audience:
- “Is it healthy right now?” (on-call): uptime per region, error rate, 5xx rate, p75 LCP/INP/CLS last hour vs last week, current release.
- “How did the last release do?” (the team): metrics split by release version, new error issues, bundle size trend.
- “How are we doing for users?” (leadership, weekly): p75 Core Web Vitals by device class and key route over 90 days, against targets and budgets (Chapters 16 and 18), with annotations for releases and incidents.
Show percentiles, not averages. Show trends with release markers. Put the target lines on the chart.
Privacy: GDPR, India’s DPDP Act
Section titled “Privacy: GDPR, India’s DPDP Act”Observability collects data about people. That’s regulated.
- GDPR (EU, since 2018) treats a lot of what monitoring collects as personal data — including identifiers like IP addresses and user IDs in many contexts. It requires a lawful basis for processing, data minimization, purpose limitation, retention limits, security, and — for non-essential tracking via cookies and similar storage — consent under the related ePrivacy rules. Penalties can reach up to 4% of global annual turnover (or a large fixed amount) for serious violations.
- India’s Digital Personal Data Protection (DPDP) Act, 2023, with its implementing rules notified in late 2025 and obligations phasing in over the following period, establishes a consent-centred regime for processing digital personal data: notice and consent (with limited “legitimate uses”), purpose limitation, data minimization, reasonable security safeguards, breach notification, rights for individuals (access, correction, erasure, grievance redressal), extra protections for children’s data, and a Data Protection Board with the power to impose significant penalties (up to hundreds of crores of rupees for the most serious failures, such as inadequate security safeguards).
This isn’t legal advice, and the details matter — involve your legal team. But the engineering principles are the same everywhere:
- Collect the minimum: you don’t need a user’s email in an error event; a pseudonymous ID is enough. You don’t need full URLs with query strings in RUM.
- Scrub: strip or hash IPs, remove tokens and personal data from URLs, breadcrumbs and logs before they leave the browser or server; mask session replays.
- Consent where needed: load non-essential analytics and replay only after consent; essential error monitoring and security logging often have a different basis — get advice.
- Retention limits: delete raw logs and events after a defined period; keep aggregates.
- Know where data goes: third-party processors, data residency, contracts.
- Self-host when it simplifies things: a self-hosted error tracker and RUM endpoint on your own infrastructure keeps data under your control.
An end-to-end stack for Lumen on a VPS
Section titled “An end-to-end stack for Lumen on a VPS”Here’s a pragmatic, modest-cost observability stack for the Next.js app you deployed to your VPS in Chapter 4. The shape matters more than the specific products:
- Errors: an error-monitoring SDK in the Next.js app (client and server), release tagged from CI, hidden source maps uploaded in CI, PII scrubbing on, replay off or masked and sampled. Hosted (Sentry or similar) or self-hosted (GlitchTip or similar) on a separate small box.
- RUM:
web-vitalswith attribution (Chapter 17) beaconing to a tiny/rumendpoint on your server, writing JSON lines; a nightly job aggregates p75 by route × device class × release into a small database; or use an RUM product. - Logs: nginx access logs in JSON with request IDs and upstream timing; the Node app logging structured JSON (pino or similar) with the same request ID; both shipped to a log store (a hosted log service, or a self-hosted stack such as Grafana Loki).
- Metrics: node and system metrics (CPU, memory, disk, Node event-loop lag, process restarts) scraped into a time-series database (e.g., Prometheus) with Grafana dashboards; or a hosted agent.
- Uptime: an external uptime service checking the homepage (with content assertion),
/api/health, certificate expiry, from three regions, every minute. - Synthetic performance: a scheduled Lighthouse or WebPageTest run on key URLs from an Indian location daily, trending results.
- Product analytics (separate concern): privacy-friendly, consent-aware, cookieless if possible.
- Alerts (to a phone app or chat channel the on-call person actually watches):
- homepage or health check failing from ≥ 2 regions for 2 minutes → page;
- JS error rate (from your code) > 2× baseline for 10 minutes after a release → page during the day, ticket at night;
- p75 INP or LCP on mobile crossing “poor” for a release cohort → ticket (and roll back if severe);
- certificate expiring in < 14 days → ticket;
- disk > 85% → ticket (before it’s 100% and an outage).
- Runbooks: for each page alert, a short doc: what it means, how to check, how to roll back (Chapter 4’s symlink swap), who to escalate to.
What you now know
Section titled “What you now know”You can map the golden signals to a frontend; run error monitoring with release tagging and private source maps; use session replay responsibly; combine RUM and synthetic monitoring; write structured, correlated logs; design uptime checks that catch “200 OK” outages; design alerts that avoid fatigue; build dashboards for different audiences; apply GDPR and DPDP-style privacy principles in engineering; and assemble an end-to-end observability stack for a VPS-hosted Next.js app.
Self-quiz
Section titled “Self-quiz”1. Why tag errors with a release version?
To see which deploy introduced an error and decide to roll back.
2. How do you get readable stack traces without publishing source code?
Generate hidden source maps, upload them privately to the error tracker in CI, don’t deploy them publicly.
3. What can synthetic monitoring catch that RUM can’t?
Outages and regressions when there’s no traffic, with a consistent baseline.
4. What makes a good page-worthy alert?
Actionable, symptom-based (user-visible), urgent, owned, with a runbook.
5. Name three data-minimization steps for RUM and error data.
Pseudonymous IDs instead of emails; strip query strings/tokens from URLs; truncate or drop IPs; mask replay; retention limits.
Exercises
Section titled “Exercises”Warm-up 1. Page or ticket? (a) Disk at 70%. (b) Checkout errors 8% for 10 minutes. (c) p75 CLS rose from 0.04 to 0.08 this week. (d) TLS certificate expires in 5 days, auto-renew failed.
Solution
(a) dashboard/ticket; (b) page; (c) ticket; (d) ticket today with high priority (page if < 48 h and renewal can’t be fixed).
Warm-up 2. Scrub it. A breadcrumb contains GET /reset-password?token=abc123&email=a@b.com. What do you send?
Solution
GET /reset-password with query parameters removed or redacted.
Warm-up 3. Green but broken. Uptime is 100% but users report blank pages. What’s your first hypothesis and the monitoring fix?
Solution
A 200 response with broken content (cached error page, failing data API, JS crash). Add content assertions and a client error-rate alert.
Lab 4. Errors for Lumen. Add an error-monitoring SDK to Lumen, tag releases in CI, upload hidden source maps, and trigger a test error. Confirm the stack trace shows original file and line.
Expected outcome
A grouped issue with a readable stack trace, breadcrumbs and release tag; no .map files reachable publicly.
Lab 5. Structured logs with request IDs. Configure nginx to log JSON with a request ID, pass it to Node, log it in the app, return it as a response header.
Expected outcome
One ID lets you find a request’s nginx line and app logs together.
Lab 6. Uptime with content checks. Set up an external uptime check for Lumen from multiple regions that asserts on homepage content and certificate expiry.
Expected outcome
Stopping your Node process (but not nginx) triggers an alert; a healthy page doesn’t.
Lab 7. The release dashboard. Build a dashboard showing p75 LCP/INP/CLS and error rate by release for Lumen, using your Chapter 17 RUM data.
Expected outcome
Each deploy appears as a cohort; regressions are visible within hours.
Staff 8. Alert review. Take your team’s alerts from the last month. Classify each firing as actionable or noise; propose changes to cut noise by half without losing coverage.
Model approach
Tabulate alerts, outcomes and time to acknowledge; delete or demote cause-based alerts; add windows and deploy-relative thresholds; merge duplicates; add runbooks.
Staff 9. Privacy design review. Write a short privacy section for your observability design: what’s collected, why, basis, minimization, retention, processors, user rights handling.
Model outline
Data inventory table; purpose per item; consent gating for non-essential; scrubbing rules in SDK config; retention per store; processor list; deletion workflow; review with legal.