22. SEO Plumbing
“The best place to hide a body is page two of Google.”— an old SEO joke that stopped being funny when people stopped visiting page one
The story: the librarian who reads everything
Section titled “The story: the librarian who reads everything”Imagine a library that wants a catalogue of every book ever printed. It employs an army of tireless readers. Each reader walks into a bookshop, reads every book it can find, follows every reference in every bibliography to other bookshops, and sends back notes. Back at the library, editors decide which books are worth listing, what each is about, and in what order to recommend them when someone asks a question.
Some bookshops put a sign on the door: “Readers welcome, but not in the back room.” Some keep a printed list of every book they stock, with the date each was last revised, to save the readers time. Some books say on their first page: “This is a reprint; the original is in the other shop.” Some have their pages glued shut until you shake them in a particular way (that’s JavaScript).
Search engines are the readers and the editors. SEO plumbing is everything you do so the readers can get in, find everything, understand what each page is and which version is the real one — and so the editors aren’t misled. It’s not the marketing side of SEO (keywords, content strategy). It’s the engineering side, and it’s yours.
Crawl, render, index
Section titled “Crawl, render, index”Google’s pipeline (others are broadly similar):
- Crawl. Googlebot discovers URLs (from links, sitemaps, previous crawls) and fetches them, respecting
robots.txtand managing crawl budget — how many URLs it’s willing to fetch from your site in a given time, based on your server’s health and the site’s perceived value. - Render. For pages that return HTTP 200, Google queues them for rendering with an evergreen headless Chromium (the Web Rendering Service), executing JavaScript to see the final DOM. This used to be described as the “two waves” of indexing — HTML first, rendered content later — because rendering can be delayed. Today Google says most pages are rendered quickly, but rendering is still a separate, resource-limited step, and content that only appears after JS runs is at the mercy of it working: JS errors, blocked resources, slow APIs and things that require interaction (clicks, scrolls) mean content Google never sees.
- Index. Google analyses the rendered page — content, links, canonicals, structured data, language — decides whether to index it, and which URL is canonical among duplicates.
- Rank. When someone searches, hundreds of signals decide the order. Relevance and content quality dominate. Page experience (including Core Web Vitals) is one input among many.
robots.txt — and what it does not do
Section titled “robots.txt — and what it does not do”robots.txt lives at the root of each host (https://lumen.example.in/robots.txt) and tells crawlers which paths they may crawl:
User-agent: *Disallow: /admin/Disallow: /search?Allow: /
Sitemap: https://lumen.example.in/sitemap.xmlWhat it does not do:
- It does not prevent indexing. A disallowed URL can still appear in search results (as a bare URL, without a snippet) if other sites link to it. To keep a page out of the index, it must be crawlable and say
noindex(below). Disallowing it makes Google unable to see thenoindex— a classic contradiction. - It is not security. It’s a public file that politely asks well-behaved bots to stay out. It lists exactly where your sensitive paths are. Protect private content with authentication.
- It doesn’t apply across hosts. Each subdomain needs its own.
- Blocking CSS/JS breaks rendering. If Googlebot can’t fetch your JS bundles, it renders an empty page. Don’t disallow
/_next/or/assets/.
The format was formally standardized in 2022 after decades as an informal convention.
sitemap.xml in full
Section titled “sitemap.xml in full”A sitemap is a file listing the URLs you want search engines to know about. It helps discovery — especially for new sites, large sites, pages with few internal links, and recently changed content.
<?xml version="1.0" encoding="UTF-8"?><urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"> <url> <loc>https://lumen.example.in/essays/monsoon/</loc> <lastmod>2026-11-02</lastmod> </url> <url> <loc>https://lumen.example.in/essays/chai/</loc> <lastmod>2026-10-18</lastmod> </url></urlset>The rules that matter:
- Absolute, canonical URLs only. Exactly the URLs you want indexed: final, 200-status, not redirected, not
noindex, matching your canonical tags (trailing slash and all). - Limits: a single sitemap file holds up to 50,000 URLs and 50 MB uncompressed. Bigger sites use a sitemap index that lists multiple sitemap files (by section:
sitemap-essays.xml,sitemap-tags.xml). lastmodhonesty. Google useslastmodif it’s consistently accurate, to decide what to recrawl. Set it to the date the page’s main content actually changed — not the build date, not “now” on every request. A sitemap where every URL claims to have changed today teaches Google to ignore yourlastmodentirely.changefreqandpriorityare ignored by Google. Skip them.- Generate it, don’t hand-write it. Next.js App Router supports a
sitemap.tsroute that returns entries from your data; Astro has an official sitemap integration (this book uses it); most frameworks have equivalents. Build-time generation for static sites; on-demand (and cached) for dynamic catalogues. - Tell search engines: reference it in
robots.txt(Sitemap:line) and submit it in Google Search Console (and Bing Webmaster Tools), where you can see how many submitted URLs were indexed and why others weren’t. The old “ping” endpoint for notifying Google of sitemap changes has been retired; Google re-reads sitemaps it knows about.
Canonicals, duplicates and parameters
Section titled “Canonicals, duplicates and parameters”The same content often lives at several URLs: /essays/monsoon, /essays/monsoon/, /essays/monsoon?utm_source=whatsapp, http://…, www.…, a print version, an AMP version. Search engines must pick one to index (the canonical); you can tell them which:
<link rel="canonical" href="https://lumen.example.in/essays/monsoon/">Rules: every indexable page has a self-referencing canonical with the absolute, preferred URL; canonical tags, internal links, sitemaps and redirects all agree on the same form (protocol, host, trailing slash); a canonical is a strong hint, not a command — if signals conflict, Google chooses for itself.
hreflang for multiple languages and regions
Section titled “hreflang for multiple languages and regions”If Lumen publishes the monsoon essay in English and Hindi, hreflang tells search engines which URL serves which audience:
<link rel="alternate" hreflang="en-IN" href="https://lumen.example.in/en/essays/monsoon/"><link rel="alternate" hreflang="hi-IN" href="https://lumen.example.in/hi/essays/monsoon/"><link rel="alternate" hreflang="x-default" href="https://lumen.example.in/essays/monsoon/">Rules: annotations must be reciprocal (each version lists all versions, including itself); use valid language (and optionally region) codes; x-default for the fallback/selector page; each URL is self-canonical. Get it wrong and the wrong language version ranks in the wrong country.
Meta robots and X-Robots-Tag
Section titled “Meta robots and X-Robots-Tag”Page-level indexing instructions:
<meta name="robots" content="noindex, follow">noindex— don’t include this page in search results (it must be crawlable for this to be seen).nofollow— don’t follow links on this page.nosnippet,max-snippet:N,max-image-preview:large— control how results look.
For non-HTML files (PDFs, images), or to apply at the server level, use the HTTP header X-Robots-Tag: noindex. Typical uses: internal search result pages, thin filter combinations, thank-you pages, staging and preview deployments (which brings us to the war story below).
Status codes and redirects
Section titled “Status codes and redirects”Search engines take HTTP semantics seriously:
- 200 — indexable content.
- 301 / 308 — permanent move. Signals consolidate to the target. Use for changed URLs, domain migrations, http→https, enforcing trailing slashes.
- 302 / 307 — temporary. The original URL stays indexed. Don’t use for permanent moves.
- 404 / 410 — gone. The URL drops out of the index over time (410 marginally faster). Fine and normal for deleted content.
- “Soft 404” — a page that returns 200 but says “not found” (common in SPAs that render a not-found component on a 200 shell). Search engines detect and treat these as errors; return a real 404.
- 5xx / 429 — server trouble. Persistent errors reduce crawling and can drop pages from the index. A short outage is fine; days of 5xx are not.
- Redirect chains (A→B→C→D) waste crawl budget and user round trips (Chapter 5). Redirect directly to the final URL.
Open Graph, Twitter cards — the share preview
Section titled “Open Graph, Twitter cards — the share preview”When someone pastes Lumen’s link into WhatsApp, Slack, X or LinkedIn, those apps fetch the page and look for Open Graph (and Twitter card) meta tags to build a preview:
<meta property="og:title" content="Monsoon on Marine Drive"><meta property="og:description" content="A photo essay on the first rain."><meta property="og:image" content="https://lumen.example.in/og/monsoon.jpg"><meta property="og:url" content="https://lumen.example.in/essays/monsoon/"><meta property="og:type" content="article"><meta name="twitter:card" content="summary_large_image">Not a ranking factor, but in a WhatsApp-first market like India, the share preview is your landing page. Use an absolute image URL, an image around 1200×630, and server-rendered tags (most link-preview bots don’t run JavaScript).
Structured data (JSON-LD)
Section titled “Structured data (JSON-LD)”Structured data describes your page’s content in a machine-readable vocabulary (schema.org), usually as a JSON-LD script:
<script type="application/ld+json">{ "@context": "https://schema.org", "@type": "Article", "headline": "Monsoon on Marine Drive", "datePublished": "2026-07-14", "author": { "@type": "Person", "name": "Lumen Editors" }, "image": ["https://lumen.example.in/og/monsoon.jpg"] }</script>Search engines use it to understand content and to show rich results (product prices and ratings, recipes, events, breadcrumbs, articles in Top Stories). Only certain types are eligible for rich results, eligibility rules change (some formerly common rich results have been restricted or retired), and markup must match the visible content. Validate with Google’s Rich Results Test. Structured data doesn’t directly boost ranking; it changes how your result looks, which affects clicks.
Core Web Vitals as a ranking signal — honestly
Section titled “Core Web Vitals as a ranking signal — honestly”Google’s page experience signals, including Core Web Vitals, became part of ranking in 2021 (mobile) and 2022 (desktop). How much do they matter? Google’s consistent message: relevance and content quality dominate; page experience is a lighter signal that can matter when results are otherwise similar. Sites don’t leap from page three to the top by fixing CLS, and a slow page with uniquely relevant content still ranks.
So the honest pitch for performance work is not “it’ll fix our SEO.” It’s: performance helps users and conversion directly (Chapter 16), and it’s a modest positive for search on top. Use field data (CrUX via Search Console) because that’s what Google uses — not your Lighthouse score.
SPA SEO pitfalls
Section titled “SPA SEO pitfalls”Client-rendered apps (Chapter 7) are where SEO plumbing most often breaks:
- Empty initial HTML. Content exists only after JS runs; rendering failures or delays mean nothing gets indexed. Server-render or statically generate indexable pages.
- Links that aren’t links. Navigation via
onClickhandlers on<div>s or buttons isn’t crawlable. Use real<a href>elements (your router’sLinkcomponent renders them). - Fragment URLs (
/#/essays/monsoon): the fragment isn’t sent to servers and is generally not treated as a separate page. Use the History API with real paths. - Soft 404s: unknown routes render “Not found” with a 200 status. Return a real 404 from the server.
- Per-route metadata: titles, descriptions, canonicals and OG tags must be in the server-rendered HTML per route, not set by client JS after load.
- Content behind interaction: tabs, “load more” buttons and infinite scroll hide content from crawlers that don’t click or scroll. Provide paginated URLs with real links.
- Blocked resources in
robots.txtbreaking rendering.
AI crawlers and robots.txt in the current era
Section titled “AI crawlers and robots.txt in the current era”Alongside search crawlers, a new class of crawlers collects content for AI systems: some for training models, some for AI search and answers (fetching pages to cite in responses), and some acting on behalf of a user’s request in real time. Major AI companies publish separate user-agent tokens for these purposes (for example, OpenAI distinguishes its training crawler from its search crawler and its user-initiated fetcher; Anthropic, Perplexity, Apple, Common Crawl and others publish their own tokens; Google offers a separate Google-Extended token that controls use of content for its AI models without affecting Search indexing).
You can allow or disallow each in robots.txt:
User-agent: GPTBotDisallow: /
User-agent: Google-ExtendedDisallow: /Considerations: blocking a training crawler doesn’t remove content already collected; blocking an AI search crawler can remove you from AI answers that would cite and link you; robots.txt is honoured voluntarily, and not every bot is well-behaved; CDNs now offer bot-management options, including blocking AI crawlers by default and charging them. Proposals like llms.txt (a curated markdown index for LLMs) exist but aren’t a standard that major search engines use. Decide deliberately, as a business choice, and revisit — this area is moving fast.
When it goes wrong: the accidental noindex
Section titled “When it goes wrong: the accidental noindex”This is a composite of a story that SEO consultants tell constantly, because it happens constantly.
A company redesigns its site. During development, the staging environment correctly carries <meta name="robots" content="noindex"> (and a Disallow: / in robots.txt) so search engines don’t index unfinished pages. Launch day: the new site goes live — and the staging configuration goes with it. The tag is in a shared layout template; the environment variable that should have switched it off was never set in production.
Nothing breaks. The site works perfectly for visitors. Over the next days and weeks, as Google recrawls, pages drop out of the index. Organic traffic slides 30%, 60%, 90%. Marketing notices a few weeks later and assumes it’s an algorithm update. Someone finally runs a URL inspection in Search Console: “Excluded by ‘noindex’ tag.” The fix takes one minute. Recovery takes weeks to months as pages are recrawled, and some lost rankings never fully return.
The variant with robots.txt is just as common: a Disallow: / carried from staging blocks all crawling; pages linger in results without snippets, then fade.
Lessons, as engineering:
- Robots directives are configuration that differs by environment; test them like code. A CI check that fails the production build if the production HTML contains
noindexor productionrobots.txtcontainsDisallow: /. - Protect staging with authentication, not just
noindex(it also stops leaks of unreleased work). - Monitor: alert on sudden drops in indexed pages or organic clicks (Search Console data), and run a synthetic check that fetches the homepage and asserts no
noindex(Chapter 23).
What you now know
Section titled “What you now know”You can explain crawl, render, index and rank and where each fails; write robots.txt correctly and know its limits; produce honest sitemaps and sitemap indexes from your framework and submit them; set canonicals and reciprocal hreflang; use meta robots and X-Robots-Tag; choose correct status codes and avoid soft 404s; build share previews and structured data; put Core Web Vitals’ ranking weight in honest perspective; avoid SPA pitfalls; make a deliberate choice about AI crawlers; and prevent the accidental noindex.
Self-quiz
Section titled “Self-quiz”1. Does Disallow in robots.txt keep a page out of search results?
No — it blocks crawling; the URL can still be indexed from links. Use crawlable noindex to exclude it.
2. What makes lastmod useful to Google?
Consistent accuracy — reflecting real content changes, not build times.
3. What’s a soft 404 and why is it bad?
A “not found” page returned with 200; search engines treat it as an error and it wastes crawl and confuses indexing.
4. Why must OG tags be server-rendered?
Most link-preview bots don’t execute JavaScript.
5. How much do Core Web Vitals affect ranking?
A modest signal; relevance and content quality dominate. The bigger payoff is user experience and conversion.
Exercises
Section titled “Exercises”Warm-up 1. Spot the contradiction. A page is Disallowed in robots.txt and has noindex. What happens?
Solution
Google can’t crawl it, so it never sees noindex; the URL may still appear in results from external links. Allow crawling so noindex works.
Warm-up 2. Pick the status. An essay was renamed; a product is permanently discontinued with no replacement; a seasonal page will return next year.
Solution
301/308 to the new URL; 404 or 410; keep it live (200) with updated content, or 302 temporarily if it must redirect.
Warm-up 3. Canonical consistency. List four places the canonical form of a URL must agree.
Solution
Canonical tag, internal links, sitemap entries, redirects (and hreflang annotations).
Lab 4. Lumen’s plumbing. Add to Lumen: a generated sitemap.xml with real lastmod from essay data, robots.txt with the sitemap line, self-canonicals, per-essay titles/descriptions, OG/Twitter tags, and Article JSON-LD.
Expected outcome
View source on an essay shows all tags in server HTML; the Rich Results Test validates the JSON-LD; the sitemap lists only canonical URLs.
Lab 5. Search Console. Verify Lumen’s domain in Search Console (DNS TXT record — Chapter 5), submit the sitemap, and inspect one URL.
Expected outcome
The URL inspection shows crawl, render (screenshot of rendered HTML) and index status.
Lab 6. The noindex guard. Add a CI check that builds production and fails if any HTML contains a robots noindex (except an allow-list) or if robots.txt disallows /. Make preview deployments send X-Robots-Tag: noindex.
Expected outcome
Staging safety without risking production; the check fails if staging config leaks.
Lab 7. Share preview. Paste Lumen’s essay URL into WhatsApp (or a link-preview debugger). Fix whatever looks wrong.
Hint
Absolute image URL, correct size, server-rendered tags; caches in preview tools may need refreshing.
Staff 8. AI crawler policy. Write a one-page recommendation for your company’s stance on AI crawlers: training vs search vs user agents, business trade-offs, and review cadence.
Model points
Distinguish purposes; consider traffic from AI citations vs value of exclusivity; legal and licensing context; implementation in robots.txt plus CDN bot management; monitoring bot traffic; quarterly review.
Staff 9. Migration plan. Plan a URL structure migration for a 20,000-page site with minimal SEO loss.
Model outline
Full URL map; one-hop 301s; update internal links, canonicals, sitemaps, hreflang; keep old sitemap briefly to speed recrawl; monitor Search Console coverage and traffic by section; staged rollout; rollback plan.