4. Continuous Delivery — Shipping Without Fear
“Deploy on Friday? Sure. Deploy something you can’t undo on Friday? Different question.”— a release engineer with a weekend
The story: the day the boarding passes were handwritten
Section titled “The story: the day the boarding passes were handwritten”On a Friday in July 2024, travellers at airports across the world — including Delhi and Mumbai — found check-in counters with staff writing boarding passes by hand. Banks couldn’t process payments, hospitals postponed procedures, broadcasters went off air. Something on the order of eight and a half million Windows machines were stuck in a crash loop, showing the blue screen of death on every boot.
The cause was a routine content update from CrowdStrike, a security vendor whose software runs deep inside Windows, at the kernel level. The update was a configuration-like data file for its threat-detection engine. A mismatch between what that file contained and what the code reading it expected caused an out-of-bounds memory read — in the kernel, where a crash takes down the whole machine. And because the update was treated as content rather than code, it went to essentially every customer at once.
Not to 1% first. Not to the vendor’s own machines for an hour. Everyone, globally, within minutes.
Recovery was brutal, because the broken machines couldn’t boot far enough to receive a fix; many needed someone to physically visit them. Airlines took days to recover; one large US carrier reported losses in the hundreds of millions of dollars.
The lesson is not “don’t write bugs.” Everyone writes bugs. The lesson is about blast radius: how many people a bad change can hurt before you notice, and how fast you can take it back. That is what Continuous Delivery is really about.
Continuous Delivery vs. Continuous Deployment
Section titled “Continuous Delivery vs. Continuous Deployment”Continuous Integration (Chapter 3) means every change is built and checked automatically. Continuous Delivery means every change that passes is deployable — packaged, versioned, and one button away from production. Continuous Deployment goes one step further: every change that passes ships automatically, no button.
The goal of all three is the same: make releases small, frequent and boring. A release with three commits in it is easy to reason about and easy to roll back. A release with three months in it is a gamble.
Environments
Section titled “Environments”Software usually passes through a ladder of environments:
- Local — your machine. Fast feedback, least realistic.
- Preview — a temporary deployment per pull request, with its own URL. Designers, PMs and reviewers click the real thing. This is one of the highest-leverage practices in frontend; once you’ve had it, you can’t go back.
- Staging — a production-like environment shared by the team, ideally with production-like data (anonymized) and the same infrastructure.
- Production — the real users.
Configuration, secrets, and the build-time trap
Section titled “Configuration, secrets, and the build-time trap”Every environment differs in configuration: API URLs, feature flags, keys. The twelve-factor rule of thumb is to keep config in the environment (environment variables) rather than in code, so the same build can run anywhere.
Two distinctions trip up frontend engineers constantly:
1. Environment variables vs. secrets. A secret is any value whose disclosure causes harm: database passwords, API keys with write access, signing keys. Secrets live in a secret store (CI secrets, a vault, the host’s encrypted env) and are injected at runtime on the server. Non-secret config (a public analytics ID, the API base URL) is just config.
2. Build-time vs. runtime variables. Server code reads process.env.X when it runs, so changing the variable and restarting changes behaviour. But browser code has no process.env. When a bundler sees process.env.NEXT_PUBLIC_API_URL or import.meta.env.VITE_API_URL in client code, it replaces that expression with the literal value at build time. The value is baked into the JavaScript file, which is public.
That’s why Next.js requires the NEXT_PUBLIC_ prefix (and Vite VITE_): it’s a deliberate speed bump that says “this value will be published to every visitor.” Put a secret there and you’ve shipped it to the world. Anyone can open DevTools, search the bundle, and read it. Rotating it afterwards is the only fix.
Deployment strategies
Section titled “Deployment strategies”How you move users from version N to N+1 determines your blast radius.
- Big-bang. Stop the old, start the new. Simple; a bad release hits everyone; often involves downtime.
- Rolling. Replace instances a few at a time. No downtime; for a while, old and new run side by side (so they must be compatible with each other and with the data).
- Blue-green. Run two complete environments. “Blue” serves traffic; deploy to “green”; test it; flip the router. Rollback is flipping back. Costs double infrastructure during the switch.
- Canary. Send a small slice of traffic (1%, then 5%, 25%, 100%) to the new version, watching error rates and metrics at each step. Automated canaries roll back on their own if metrics degrade. This is the strategy CrowdStrike’s content updates lacked.
- Feature flags. Deploy code dark — present but switched off — and release it separately by turning a flag on for some users. Decouples deploy (a technical event) from release (a product event).
- Dark launch. Run new code paths in production on real traffic without showing results (e.g., call the new search backend in the background and compare), to test load and correctness invisibly.
Rollbacks and immutable deploys
Section titled “Rollbacks and immutable deploys”The single most important property of a deploy system is how fast you can go back. The best way to make that fast is immutable deploys: every deploy produces a new, versioned, never-modified artifact (a folder, a container image, a set of hashed files on a CDN). Going back is pointing traffic at the previous artifact — seconds, not a rebuild.
Contrast with “SSH in, git pull, rebuild, restart”: the old version is gone the moment you pull, the build might fail halfway, and rolling back means rebuilding the old code under pressure.
The hard part: data
Section titled “The hard part: data”Code rolls back in seconds. Data doesn’t. If version N+1 renames a database column and you roll back to N, version N looks for a column that no longer exists. That’s why database migrations are the genuinely hard part of continuous delivery, and the standard answer is expand/contract (also called parallel change):
- Expand: add the new column alongside the old. Deploy code that writes both, reads old.
- Migrate: backfill the new column.
- Switch: deploy code that reads new (still writes both).
- Contract: once nothing reads the old column, stop writing it and drop it.
Each step is independently deployable and independently reversible. It’s slower and more boring than one big migration. Boring is the point.
Two pipelines, conceptually
Section titled “Two pipelines, conceptually”You’ll set these up for Lumen in the labs. Here’s the shape of each and why each part exists.
(a) A Next.js app on your own VPS
Section titled “(a) A Next.js app on your own VPS”A VPS is a rented Linux machine. On it you need: something that runs your Node server and restarts it if it crashes, something in front of it that handles HTTPS and serves static files efficiently, and a way to get new versions there safely.
- Build in CI, not on the server. Building on a small VPS is slow, needs all your dev dependencies installed in production, and fails halfway when memory runs out. CI builds the artifact (or a Docker image) once.
- Ship it with a deploy key. CI authenticates to the VPS with an SSH key that can do only one thing (deploy), not your personal key.
- Release directories + symlink swap. Upload each release to
/srv/lumen/releases/2026-11-20-1430/, then atomically point acurrentsymlink at it and reload. Rollback = point the symlink at the previous folder. This gives you immutable deploys on a single machine. - A process manager. PM2 or systemd keeps the Node process alive, restarts it on crash, and can do graceful reloads (start new workers, then stop old ones) for near-zero downtime.
- Docker (optional). Packages the app with its exact runtime. The “artifact” becomes an image tagged by commit; rollback is running the previous tag.
- nginx as a reverse proxy. Terminates TLS, serves
/_next/staticfiles directly with long cache headers, compresses, and forwards everything else to Node onlocalhost:3000. Node is good at running your app; nginx is good at being a web server. - HTTPS via Let’s Encrypt. A free, automated certificate authority. Certbot proves you control the domain, fetches a certificate, configures nginx, and renews it automatically before it expires (the lifetimes are short — and getting shorter — precisely to force automation).
(b) A static site on a CDN with previews
Section titled “(b) A static site on a CDN with previews”For a static or statically-exported site, the deploy target is a CDN platform (Cloudflare, Netlify, Vercel, and others). The pipeline is simpler and more powerful:
- CI (or the platform itself) builds on every push.
- Each build is uploaded as an immutable version with its own URL.
- Pull requests get preview URLs automatically, posted as a PR comment.
- Merging to main promotes a version to the production domain. Rollback is promoting the previous version — instant, because the files are already on the edge.
This is the gold standard of frontend delivery: every change visible at a real URL before merge, every deploy atomic, every rollback a click. This very book is deployed this way (Appendix E).
When it goes wrong
Section titled “When it goes wrong”CrowdStrike (2024) — no staged rollout for a kernel-level change; global blast radius within minutes; fix couldn’t be delivered to machines that couldn’t boot. Lesson: everything that changes production behaviour is a deploy, including “content” and “config,” and deserves canaries.
GitLab (2017). During a stressful night fighting a database replication problem, an engineer ran a delete command intended for the secondary database server on the primary. Roughly 300 GB of production data began disappearing before they stopped it. Then came the real horror: when they reached for backups, they found that several of their backup mechanisms had been silently failing or were misconfigured — a version mismatch here, an alert email being rejected there. They recovered from a snapshot that happened to have been taken about six hours earlier, losing that window of data, and live-streamed the recovery to thousands of viewers. Lesson: a backup you’ve never restored is a hope, not a backup. Test restores on a schedule.
Smaller classics: secrets shipped in a client bundle via a public env var; a migration that dropped a column while old instances still read it; a deploy script that did rm -rf $DEPLOY_DIR/ with the variable unset.
What you now know
Section titled “What you now know”You can explain environments and preview deploys, the difference between secrets and config and why NEXT_PUBLIC_ exists, compare big-bang, rolling, blue-green, canary and flags by blast radius, design immutable deploys with instant rollback, migrate data without breaking rollback, and sketch both a VPS and a CDN pipeline. You also have two of the most important incident stories in modern ops in your pocket.
Self-quiz
Section titled “Self-quiz”1. What’s the difference between deploy and release?
Deploy puts code on servers; release exposes behaviour to users (often via flags). Separating them shrinks risk.
2. Why is a NEXT_PUBLIC_ secret a leak?
Public variables are inlined into client bundles at build time; anyone can read the bundle.
3. What makes rollback fast?
Immutable, versioned artifacts that are still present — rollback is repointing traffic, not rebuilding.
4. Why can’t you just rename a column in one deploy?
During and after the deploy, old code (or a rollback) still expects the old column; expand/contract keeps both valid.
5. Why build in CI instead of on the VPS?
Consistency, speed, no dev dependencies or build failures on production, and the same artifact can be deployed and rolled back.
Exercises
Section titled “Exercises”Warm-up 1. Classify. Which are secrets: Stripe publishable key, Stripe secret key, Sentry DSN, database URL, Google Analytics ID?
Solution
Secrets: Stripe secret key, database URL. The publishable key, the Sentry DSN and the GA ID are designed to be public.
Warm-up 2. Pick a strategy. You’re changing checkout pricing logic for an e-commerce site. Which deployment strategy and why?
Solution
Feature flag + canary: release to a small percentage, compare conversion and error rates, expand gradually; instant off-switch.
Warm-up 3. Spot the danger. A deploy script runs git pull && npm install && npm run build && pm2 restart all on the server. List three risks.
Solution
Build failure leaves a half-updated state; no artifact to roll back to; install runs untrusted scripts on prod; downtime during restart; drift between machines.
Lab 4. Lumen becomes Next.js. Migrate Lumen to a Next.js App Router project. Keep the gallery and essay pages.
Expected outcome
next build shows the essay pages as static (○/●). You’ll choose rendering strategies deliberately in Chapter 7.
Lab 5. Preview deploys. Connect Lumen’s repo to a CDN platform with per-PR preview URLs. Open a PR and share the preview link with a friend.
Expected outcome
Every PR gets a unique URL; merging promotes to production; you can roll back to any previous deployment.
Lab 6. VPS with symlink releases. Deploy Lumen to a VPS behind nginx with HTTPS, using timestamped release folders and a current symlink. Practise a rollback.
Hint
Time the rollback. It should be seconds: swap symlink, reload the process manager.
Lab 7. Restore drill. Back up whatever data Lumen has (even a JSON file or SQLite database), delete it, and restore from backup. Write down how long it took.
The point
GitLab learned this lesson in production. You’re learning it on a toy.
Staff 8. Blast radius review. List every way production behaviour can change in your team’s system (code, config, flags, CMS content, third-party scripts, DNS). For each: is it staged? Can it be rolled back? How fast?
Model approach
A table with change type, owner, rollout mechanism, rollback time, monitoring. The surprising rows (CMS content, tag managers, DNS) are usually the ones with no staging and no rollback.
Staff 9. Friday deploys. Your org bans Friday deploys. Argue for or against in a half page, staff-engineer style.
Model answer
The ban treats symptoms: fear of deploys comes from big, irreversible releases. Invest in small changes, canaries, flags and fast rollback until a Friday deploy is as boring as a Tuesday one; keep a freeze only for genuinely irreversible or high-risk changes and when on-call coverage is thin.