Skip to content

4. Continuous Delivery — Shipping Without Fear

“Deploy on Friday? Sure. Deploy something you can’t undo on Friday? Different question.”— a release engineer with a weekend

The story: the day the boarding passes were handwritten

Section titled “The story: the day the boarding passes were handwritten”

On a Friday in July 2024, travellers at airports across the world — including Delhi and Mumbai — found check-in counters with staff writing boarding passes by hand. Banks couldn’t process payments, hospitals postponed procedures, broadcasters went off air. Something on the order of eight and a half million Windows machines were stuck in a crash loop, showing the blue screen of death on every boot.

The cause was a routine content update from CrowdStrike, a security vendor whose software runs deep inside Windows, at the kernel level. The update was a configuration-like data file for its threat-detection engine. A mismatch between what that file contained and what the code reading it expected caused an out-of-bounds memory read — in the kernel, where a crash takes down the whole machine. And because the update was treated as content rather than code, it went to essentially every customer at once.

Not to 1% first. Not to the vendor’s own machines for an hour. Everyone, globally, within minutes.

Recovery was brutal, because the broken machines couldn’t boot far enough to receive a fix; many needed someone to physically visit them. Airlines took days to recover; one large US carrier reported losses in the hundreds of millions of dollars.

The lesson is not “don’t write bugs.” Everyone writes bugs. The lesson is about blast radius: how many people a bad change can hurt before you notice, and how fast you can take it back. That is what Continuous Delivery is really about.

Continuous Delivery vs. Continuous Deployment

Section titled “Continuous Delivery vs. Continuous Deployment”

Continuous Integration (Chapter 3) means every change is built and checked automatically. Continuous Delivery means every change that passes is deployable — packaged, versioned, and one button away from production. Continuous Deployment goes one step further: every change that passes ships automatically, no button.

The goal of all three is the same: make releases small, frequent and boring. A release with three commits in it is easy to reason about and easy to roll back. A release with three months in it is a gamble.

Software usually passes through a ladder of environments:

  • Local — your machine. Fast feedback, least realistic.
  • Preview — a temporary deployment per pull request, with its own URL. Designers, PMs and reviewers click the real thing. This is one of the highest-leverage practices in frontend; once you’ve had it, you can’t go back.
  • Staging — a production-like environment shared by the team, ideally with production-like data (anonymized) and the same infrastructure.
  • Production — the real users.
Environment ladderFour steps rising left to right: local, preview per pull request, staging, production. An arrow labelled same artifact runs along the steps. Annotations: realism increases, blast radius increases.LocalPreview (per PR)StagingProductionrealism ↑ · blast radius ↑same build artifact climbs every step
Figure 4.1. The environment ladder. Realism increases upwards; so does the cost of a mistake. The same build artifact should climb the ladder — never rebuild per environment if you can help it.

Configuration, secrets, and the build-time trap

Section titled “Configuration, secrets, and the build-time trap”

Every environment differs in configuration: API URLs, feature flags, keys. The twelve-factor rule of thumb is to keep config in the environment (environment variables) rather than in code, so the same build can run anywhere.

Two distinctions trip up frontend engineers constantly:

1. Environment variables vs. secrets. A secret is any value whose disclosure causes harm: database passwords, API keys with write access, signing keys. Secrets live in a secret store (CI secrets, a vault, the host’s encrypted env) and are injected at runtime on the server. Non-secret config (a public analytics ID, the API base URL) is just config.

2. Build-time vs. runtime variables. Server code reads process.env.X when it runs, so changing the variable and restarting changes behaviour. But browser code has no process.env. When a bundler sees process.env.NEXT_PUBLIC_API_URL or import.meta.env.VITE_API_URL in client code, it replaces that expression with the literal value at build time. The value is baked into the JavaScript file, which is public.

That’s why Next.js requires the NEXT_PUBLIC_ prefix (and Vite VITE_): it’s a deliberate speed bump that says “this value will be published to every visitor.” Put a secret there and you’ve shipped it to the world. Anyone can open DevTools, search the bundle, and read it. Rotating it afterwards is the only fix.

Build-time vs runtime environment variablesAt build time, NEXT_PUBLIC_API_URL is substituted into the client bundle as a literal string, which is served publicly to every browser. At runtime, DATABASE_URL is read by the server process from its environment and never leaves the server.NEXT_PUBLIC_API_URL= https://api.lumen…buildinlines literalclient.js: fetch(“https://api…”)public: every visitor can read itDATABASE_URL(secret store)server processreads at runtimenever in any bundle;change + restart, no rebuild
Figure 4.2. Build-time vs runtime. Public client variables are inlined into the bundle at build time — anyone can read them. Server variables stay on the server and can change without rebuilding.

How you move users from version N to N+1 determines your blast radius.

  • Big-bang. Stop the old, start the new. Simple; a bad release hits everyone; often involves downtime.
  • Rolling. Replace instances a few at a time. No downtime; for a while, old and new run side by side (so they must be compatible with each other and with the data).
  • Blue-green. Run two complete environments. “Blue” serves traffic; deploy to “green”; test it; flip the router. Rollback is flipping back. Costs double infrastructure during the switch.
  • Canary. Send a small slice of traffic (1%, then 5%, 25%, 100%) to the new version, watching error rates and metrics at each step. Automated canaries roll back on their own if metrics degrade. This is the strategy CrowdStrike’s content updates lacked.
  • Feature flags. Deploy code dark — present but switched off — and release it separately by turning a flag on for some users. Decouples deploy (a technical event) from release (a product event).
  • Dark launch. Run new code paths in production on real traffic without showing results (e.g., call the new search backend in the background and compare), to test load and correctness invisibly.
Deployment strategiesThree rows. Big-bang: all users switch at once to the new version, a single red block. Blue-green: a router flips from blue to green environment; rollback flips back. Canary: traffic to the new version steps from 1 percent to 5, 25 and 100 percent, with a metrics check between each step and an automatic rollback arrow.Big-bangv1 (all users)v2 (all users, at once)Blue-greenblue (v1)green (v2)routerflip = deploy; flip back = rollbackCanary1%5%25%100%checkcheckcheckmetrics bad → auto rollback
Figure 4.3. Blast radius by strategy. Canary and flags let a bad change hurt a few users for a few minutes, instead of everyone until someone notices.

The single most important property of a deploy system is how fast you can go back. The best way to make that fast is immutable deploys: every deploy produces a new, versioned, never-modified artifact (a folder, a container image, a set of hashed files on a CDN). Going back is pointing traffic at the previous artifact — seconds, not a rebuild.

Contrast with “SSH in, git pull, rebuild, restart”: the old version is gone the moment you pull, the build might fail halfway, and rolling back means rebuilding the old code under pressure.

Code rolls back in seconds. Data doesn’t. If version N+1 renames a database column and you roll back to N, version N looks for a column that no longer exists. That’s why database migrations are the genuinely hard part of continuous delivery, and the standard answer is expand/contract (also called parallel change):

  1. Expand: add the new column alongside the old. Deploy code that writes both, reads old.
  2. Migrate: backfill the new column.
  3. Switch: deploy code that reads new (still writes both).
  4. Contract: once nothing reads the old column, stop writing it and drop it.

Each step is independently deployable and independently reversible. It’s slower and more boring than one big migration. Boring is the point.

Expand and contract migrationFour steps in sequence: expand by adding a new column and writing both; migrate by backfilling; switch reads to the new column; contract by dropping the old column. Each step is labelled reversible except the final drop.1 Expandadd new, write both2 Migratebackfill3 Switchread new4 Contractdrop old (later)steps 1–3 can each be rolled back safely; step 4 only after nothing reads the old column
Figure 4.4. Expand/contract: change a schema in reversible steps, so any single deploy can be rolled back without breaking the data.

You’ll set these up for Lumen in the labs. Here’s the shape of each and why each part exists.

A VPS is a rented Linux machine. On it you need: something that runs your Node server and restarts it if it crashes, something in front of it that handles HTTPS and serves static files efficiently, and a way to get new versions there safely.

  • Build in CI, not on the server. Building on a small VPS is slow, needs all your dev dependencies installed in production, and fails halfway when memory runs out. CI builds the artifact (or a Docker image) once.
  • Ship it with a deploy key. CI authenticates to the VPS with an SSH key that can do only one thing (deploy), not your personal key.
  • Release directories + symlink swap. Upload each release to /srv/lumen/releases/2026-11-20-1430/, then atomically point a current symlink at it and reload. Rollback = point the symlink at the previous folder. This gives you immutable deploys on a single machine.
  • A process manager. PM2 or systemd keeps the Node process alive, restarts it on crash, and can do graceful reloads (start new workers, then stop old ones) for near-zero downtime.
  • Docker (optional). Packages the app with its exact runtime. The “artifact” becomes an image tagged by commit; rollback is running the previous tag.
  • nginx as a reverse proxy. Terminates TLS, serves /_next/static files directly with long cache headers, compresses, and forwards everything else to Node on localhost:3000. Node is good at running your app; nginx is good at being a web server.
  • HTTPS via Let’s Encrypt. A free, automated certificate authority. Certbot proves you control the domain, fetches a certificate, configures nginx, and renews it automatically before it expires (the lifetimes are short — and getting shorter — precisely to force automation).
VPS deploy architectureCI builds and uploads over SSH to a VPS. On the VPS, a releases folder contains several timestamped releases; a current symlink points at the newest. nginx terminates HTTPS from the internet, serves static files, and proxies to a Node process managed by PM2 or systemd, which runs from the current symlink. Let’s Encrypt renews certificates.CIbuild artifactsshVPSreleases/…-1200/…-1430/ ←current →Node (PM2/systemd)nginxTLS · static · proxyLet’s Encryptauto-renew
Figure 4.5. A single-VPS deploy that still gets immutable releases and instant rollback: CI builds, uploads a new release folder, swaps a symlink, and gracefully reloads.

For a static or statically-exported site, the deploy target is a CDN platform (Cloudflare, Netlify, Vercel, and others). The pipeline is simpler and more powerful:

  • CI (or the platform itself) builds on every push.
  • Each build is uploaded as an immutable version with its own URL.
  • Pull requests get preview URLs automatically, posted as a PR comment.
  • Merging to main promotes a version to the production domain. Rollback is promoting the previous version — instant, because the files are already on the edge.

This is the gold standard of frontend delivery: every change visible at a real URL before merge, every deploy atomic, every rollback a click. This very book is deployed this way (Appendix E).

CrowdStrike (2024) — no staged rollout for a kernel-level change; global blast radius within minutes; fix couldn’t be delivered to machines that couldn’t boot. Lesson: everything that changes production behaviour is a deploy, including “content” and “config,” and deserves canaries.

GitLab (2017). During a stressful night fighting a database replication problem, an engineer ran a delete command intended for the secondary database server on the primary. Roughly 300 GB of production data began disappearing before they stopped it. Then came the real horror: when they reached for backups, they found that several of their backup mechanisms had been silently failing or were misconfigured — a version mismatch here, an alert email being rejected there. They recovered from a snapshot that happened to have been taken about six hours earlier, losing that window of data, and live-streamed the recovery to thousands of viewers. Lesson: a backup you’ve never restored is a hope, not a backup. Test restores on a schedule.

Smaller classics: secrets shipped in a client bundle via a public env var; a migration that dropped a column while old instances still read it; a deploy script that did rm -rf $DEPLOY_DIR/ with the variable unset.

You can explain environments and preview deploys, the difference between secrets and config and why NEXT_PUBLIC_ exists, compare big-bang, rolling, blue-green, canary and flags by blast radius, design immutable deploys with instant rollback, migrate data without breaking rollback, and sketch both a VPS and a CDN pipeline. You also have two of the most important incident stories in modern ops in your pocket.

1. What’s the difference between deploy and release?

Deploy puts code on servers; release exposes behaviour to users (often via flags). Separating them shrinks risk.

2. Why is a NEXT_PUBLIC_ secret a leak?

Public variables are inlined into client bundles at build time; anyone can read the bundle.

3. What makes rollback fast?

Immutable, versioned artifacts that are still present — rollback is repointing traffic, not rebuilding.

4. Why can’t you just rename a column in one deploy?

During and after the deploy, old code (or a rollback) still expects the old column; expand/contract keeps both valid.

5. Why build in CI instead of on the VPS?

Consistency, speed, no dev dependencies or build failures on production, and the same artifact can be deployed and rolled back.

Warm-up 1. Classify. Which are secrets: Stripe publishable key, Stripe secret key, Sentry DSN, database URL, Google Analytics ID?

Solution

Secrets: Stripe secret key, database URL. The publishable key, the Sentry DSN and the GA ID are designed to be public.

Warm-up 2. Pick a strategy. You’re changing checkout pricing logic for an e-commerce site. Which deployment strategy and why?

Solution

Feature flag + canary: release to a small percentage, compare conversion and error rates, expand gradually; instant off-switch.

Warm-up 3. Spot the danger. A deploy script runs git pull && npm install && npm run build && pm2 restart all on the server. List three risks.

Solution

Build failure leaves a half-updated state; no artifact to roll back to; install runs untrusted scripts on prod; downtime during restart; drift between machines.

Lab 4. Lumen becomes Next.js. Migrate Lumen to a Next.js App Router project. Keep the gallery and essay pages.

Expected outcome

next build shows the essay pages as static (○/●). You’ll choose rendering strategies deliberately in Chapter 7.

Lab 5. Preview deploys. Connect Lumen’s repo to a CDN platform with per-PR preview URLs. Open a PR and share the preview link with a friend.

Expected outcome

Every PR gets a unique URL; merging promotes to production; you can roll back to any previous deployment.

Lab 6. VPS with symlink releases. Deploy Lumen to a VPS behind nginx with HTTPS, using timestamped release folders and a current symlink. Practise a rollback.

Hint

Time the rollback. It should be seconds: swap symlink, reload the process manager.

Lab 7. Restore drill. Back up whatever data Lumen has (even a JSON file or SQLite database), delete it, and restore from backup. Write down how long it took.

The point

GitLab learned this lesson in production. You’re learning it on a toy.

Staff 8. Blast radius review. List every way production behaviour can change in your team’s system (code, config, flags, CMS content, third-party scripts, DNS). For each: is it staged? Can it be rolled back? How fast?

Model approach

A table with change type, owner, rollout mechanism, rollback time, monitoring. The surprising rows (CMS content, tag managers, DNS) are usually the ones with no staging and no rollback.

Staff 9. Friday deploys. Your org bans Friday deploys. Argue for or against in a half page, staff-engineer style.

Model answer

The ban treats symptoms: fear of deploys comes from big, irreversible releases. Invest in small changes, canaries, flags and fast rollback until a Friday deploy is as boring as a Tuesday one; keep a freeze only for genuinely irreversible or high-risk changes and when on-call coverage is thin.