3. Version Control Workflows and Continuous Integration
“It works on my machine.” “Great, we’ll ship your machine.”— the origin story of CI, approximately
The story: forty-five minutes
Section titled “The story: forty-five minutes”On the morning of 1 August 2012, Knight Capital — one of the largest market makers on the US stock exchanges — started trading at 9:30 a.m. as usual. By about 10:15 it had lost roughly $440–460 million. The firm, which had been worth more than that, needed a rescue within days and was soon absorbed by a competitor.
What happened was not a hack. It was a deployment.
Knight was rolling out new trading code to its servers. The new code reused a configuration flag that, years earlier, had switched on an old, long-dead feature. The deploy was done by hand, to eight servers. A technician updated seven. On the eighth, the old code was still there, and when the flag was turned on, that server woke up the dead feature — which began firing off a torrent of real orders into the market, buying high and selling low, millions of times. There was no automated check that all servers had the same build. There was no kill switch anyone could find quickly. Alerts that did fire went to email.
Every single practice in this chapter and the next — identical automated builds, checks that run every time, the same artifact on every machine, fast rollback, deleting dead code instead of leaving it behind a flag — is a scar from incidents like this one. You will probably never lose half a billion dollars in forty-five minutes. But the mechanism that cost Knight its company is exactly the mechanism behind your team’s “why is staging different from production?” afternoons.
Git, as a graph you can draw
Section titled “Git, as a graph you can draw”Most developers use Git through about eight commands and a sense of dread. The dread goes away when you see what Git actually stores. It’s simpler than the commands suggest.
Git is a content-addressed store of snapshots. Four kinds of objects:
- Blob — the contents of one file. Its ID is a hash of the contents. Same contents, same ID, stored once.
- Tree — a directory listing: names → blob IDs and sub-tree IDs.
- Commit — a pointer to one root tree (a full snapshot of the project), plus parent commit(s), author, message.
- Ref — a name pointing at a commit. A branch is just a ref that moves when you commit.
HEADis a ref to the ref you’re on. Tags are refs that don’t move.
That’s it. A commit is not a diff; it’s a snapshot (efficient because unchanged files reuse the same blobs). History is a graph of commits pointing at their parents.
With that picture, the scary commands become obvious:
- Merge creates a new commit with two parents, joining two lines of history. Nothing is rewritten.
- Rebase takes your commits, and replays them one by one on top of another commit, creating new commits with new IDs. The old ones still exist until garbage-collected. That’s why rebasing a branch others have pulled is rude: you’ve replaced commits they’re building on.
- Revert creates a new commit that applies the inverse of an old one. History is preserved; the effect is undone. This is the safe “undo” for shared branches.
- Reset moves a ref to point somewhere else. Commits you “lost” are still in the reflog for a while.
- Cherry-pick copies one commit’s change onto another place, as a new commit.
Branching strategies: why everyone converged
Section titled “Branching strategies: why everyone converged”GitFlow (popularized around 2010) used long-lived develop and master branches, plus feature, release and hotfix branches. It fit software shipped as versioned releases a few times a year. For web apps deployed many times a day, it mostly produced merge pain: long-lived branches drift apart, and merging them becomes a project of its own.
Trunk-based development keeps one main branch (the trunk), always releasable. Work happens in short-lived branches — hours to a couple of days — merged via small PRs. Unfinished features hide behind feature flags (Chapter 4) instead of living on a branch for weeks.
Why did good teams converge on short-lived branches? Because merge cost grows faster than branch age. Two branches that diverged for a day conflict a little; for a month, they conflict everywhere, and the conflicts are semantic, not just textual. Small PRs are also reviewable: a reviewer can genuinely reason about 150 lines, and will rubber-stamp 3,000.
PR hygiene and review as a gate
Section titled “PR hygiene and review as a gate”A good PR is small, does one thing, and explains why in its description. It includes evidence: a screenshot for UI, a before/after number for performance, the route table diff from Chapter 1 for bundle changes. Code review catches bugs, but its bigger value is spreading knowledge and keeping the codebase coherent. Performance is a reviewable property too: “this adds a 70 KB dependency to the homepage” is a legitimate review comment, and automation (below) should surface it so reviewers don’t have to remember.
What CI actually is
Section titled “What CI actually is”Strip the vendor branding and Continuous Integration is a simple, powerful idea:
On every push, a fresh, empty computer checks out your code, installs exactly the locked dependencies, and runs every check. If any check fails, the change doesn’t merge.
The “fresh, empty” part is the point. Your laptop has years of accumulated state: global packages, a cached build, an environment variable you set in 2023, a file you forgot to commit. A CI runner has none of that. If it builds there, it builds from what’s in the repository. That’s the opposite of “works on my machine,” and it’s the cure for the Knight Capital class of problem: the artifact is produced the same way, every time, by a machine.
The vocabulary, using GitHub Actions as the example (GitLab CI, CircleCI and others use different words for the same shapes):
- Workflow — a YAML file describing what runs on which events (push, pull request, schedule).
- Runner — the machine (usually a fresh VM) that executes a job.
- Job — a set of steps on one runner. Jobs run in parallel unless one
needsanother. - Step — one command or one reusable action.
- Cache — files saved between runs keyed by something (like the lockfile hash) to speed up installs.
- Artifact — files produced by a job and kept (a build, a report) for later jobs or humans.
- Matrix — run the same job across combinations (Node 20 and 22; Ubuntu and Windows).
- Secrets — encrypted values injected as environment variables, never printed in logs.
A pipeline, read like prose
Section titled “A pipeline, read like prose”You don’t need to memorize YAML; you need to be able to read a pipeline and know what each part buys you. Here is a compact, representative workflow for a Next.js app, trimmed to the parts worth understanding:
name: cion: pull_request: # run on every PR... push: branches: [main] # ...and on every merge to main
concurrency: # a new push cancels the stale run for the same branch group: ci-${{ github.ref }} cancel-in-progress: true
jobs: quality: runs-on: ubuntu-latest # a fresh VM, every time steps: - uses: actions/checkout@v4 # get the commit - uses: pnpm/action-setup@v4 # the package manager in package.json - uses: actions/setup-node@v4 with: { node-version: 22, cache: pnpm } # cache keyed on the lockfile - run: pnpm install --frozen-lockfile # exactly the lockfile, or fail - run: pnpm lint - run: pnpm typecheck # tsc --noEmit: the bundler won't do this (Ch 1) - run: pnpm test
build: runs-on: ubuntu-latest steps: # …same setup… - run: pnpm build - run: pnpm size-limit # fail if bundles exceed the budget (Ch 18) - run: pnpm lhci autorun # Lighthouse against the built app (Ch 16)What each idea buys you:
- Triggers (
on:) decide when checks run. PRs and main, at minimum. - Concurrency saves money and time by cancelling obsolete runs.
- Fresh runner + checkout + frozen install is the anti-“works on my machine” core.
- Caching keyed on the lockfile makes installs fast without risking staleness: change the lockfile, the key changes, the cache misses.
- Lint, typecheck, test are cheap, fast and catch most mistakes. Order them fastest-first so failures arrive quickly.
- Build proves the production build works (dev and prod differ; Chapter 1).
- Size check and Lighthouse CI turn performance from a vibe into a gate. Chapter 18 sets the budgets. Lighthouse in CI is lab data on a cloud VM — noisy and not your users — so treat it as a regression alarm, not a score to celebrate.
Secrets, permissions and the supply chain (again)
Section titled “Secrets, permissions and the supply chain (again)”Remember Chapter 2’s install-time malware. In CI, pnpm install runs install scripts on a machine that may hold your deploy keys. So:
- Give the workflow’s token minimal permissions by default (read-only), and grant more per job.
- Don’t expose deploy secrets to jobs that run on pull requests from forks — those PRs contain code you haven’t reviewed.
- Separate “build” (runs untrusted dependency code) from “deploy” (holds keys): the deploy job consumes the build’s artifact and never runs
install. - Prefer short-lived credentials via OIDC (your CI proves its identity to the cloud and gets a temporary token) over long-lived secrets pasted into settings.
Flaky tests, speed and the human side
Section titled “Flaky tests, speed and the human side”A flaky test passes and fails on the same code. Flakes are worse than no tests: they teach the team to click “re-run” without reading, and one day the red is real. Common causes: timing (setTimeout in tests, animations), shared state between tests, network calls, order dependence, dates and time zones. Treat flakes as bugs: quarantine, fix, or delete.
CI speed is a productivity multiplier. If CI takes 25 minutes, developers batch changes, context-switch, and stop running it locally. Levers: caching, parallel jobs, only running affected packages in a monorepo, splitting slow end-to-end tests to run on merge rather than every push. A good target for the PR feedback loop is under ten minutes.
Required checks and branch protection make CI binding: the main branch rejects merges unless specific checks pass and a review is approved. Without them, CI is advice.
When it goes wrong
Section titled “When it goes wrong”Knight Capital is the archetype: a manual, partial deploy; a reused flag; dead code still present; no automated consistency check. Map each to a practice:
| Knight’s failure | Modern practice |
|---|---|
| Manual deploy to 8 servers, one missed | Build once in CI, deploy the same artifact everywhere, verify versions |
| Repurposed an old flag | Delete dead code; never reuse flag names |
| Dead feature still in the binary | Code removal as part of finishing a feature |
| No fast stop | Kill switches and one-command rollback (Chapter 4) |
| Alerts by email nobody read | Actionable alerts to on-call humans (Chapter 23) |
Smaller, more common failures: CI that’s green because tests didn’t run (a misconfigured glob matched zero files); caches that restore stale build output; a “temporary” continue-on-error: true that’s been there for a year.
What you now know
Section titled “What you now know”You can draw Git’s object model and explain merge, rebase and revert as graph operations; argue for trunk-based development; describe every part of a CI pipeline and what it buys; secure it against the supply chain; and connect Knight Capital’s loss to specific modern practices. You could set up a team’s CI from scratch and defend each check.
Self-quiz
Section titled “Self-quiz”1. Is a Git commit a diff or a snapshot?
A snapshot (a pointer to a full tree), stored efficiently because unchanged files share blobs.
2. Why is rebasing a shared branch discouraged?
Rebase creates new commits with new IDs; others’ work built on the old commits now diverges.
3. What makes CI “continuous integration” rather than “a server that runs tests”?
It runs on every change, from a clean environment, and gates merging, so integration problems surface within minutes.
4. Why key a dependency cache on the lockfile hash?
When dependencies change, the key changes and the cache misses, so you never install from a stale cache.
5. Why separate build and deploy jobs?
The build runs untrusted dependency code; the deploy holds production credentials. Separation limits what malicious install scripts can reach.
Exercises
Section titled “Exercises”Warm-up 1. Draw it. Draw the commit graph after: commit A on main; branch feat; commit B on feat; commit C on main; merge feat into main.
Solution
A ← C ← M on main; A ← B on feat; M has parents C and B.
Warm-up 2. Undo safely. A bad commit is already on main and others have pulled. Which command undoes it?
Solution
git revert <sha> — adds an inverse commit; doesn’t rewrite shared history.
Warm-up 3. Flake hunt. A test passes locally and fails in CI one time in ten. Name three likely causes.
Solution
Timing assumptions (slower CI CPU), dependence on test order or shared state, time zone/date differences, network calls.
Lab 4. Lumen on GitHub. Push Lumen to a new repo with a protected main requiring one review and passing checks.
Expected outcome
A direct push to main is rejected; merges require a green PR.
Lab 5. The quality job. Add a workflow that installs with a frozen lockfile, lints, type-checks and tests. Introduce a type error and confirm the PR goes red.
Hint
The type error should fail typecheck even though build (with a type-stripping bundler) would pass. That’s Chapter 1’s gotcha, now enforced.
Lab 6. Make it fast. Measure your CI duration, then add dependency caching and concurrency cancellation. Measure again.
Expected outcome
Install time drops substantially on cache hits; superseded runs are cancelled.
Lab 7. A size gate. Add a bundle-size check with a budget 10% above Lumen’s current size. Add a large dependency in a PR and watch it fail.
Hint
Tools like size-limit read built files and compare against a configured limit; make the CI step fail on breach.
Staff 8. Postmortem mapping. For an incident at your own company (or Knight Capital), write a table mapping each contributing factor to the practice that would have prevented or limited it, and who owns adopting it.
Model approach
Blameless framing; factors are systemic (missing automation, missing checks), not “the technician”; each row has an owner and a date.
Staff 9. The CI budget conversation. CI takes 28 minutes and leadership asks whether to buy bigger runners or invest engineering time. Frame the trade-off with numbers.
Model answer
Engineers × PRs/day × wait minutes × context-switch cost gives hours lost per week; compare to runner cost and to the engineering time to parallelize and cache. Usually: cheap wins (cache, parallelize, move e2e to merge) first, then bigger runners for the remaining critical path.