Skip to content

22. SEO Plumbing

“The best place to hide a body is page two of Google.”— an old SEO joke that stopped being funny when people stopped visiting page one

The story: the librarian who reads everything

Section titled “The story: the librarian who reads everything”

Imagine a library that wants a catalogue of every book ever printed. It employs an army of tireless readers. Each reader walks into a bookshop, reads every book it can find, follows every reference in every bibliography to other bookshops, and sends back notes. Back at the library, editors decide which books are worth listing, what each is about, and in what order to recommend them when someone asks a question.

Some bookshops put a sign on the door: “Readers welcome, but not in the back room.” Some keep a printed list of every book they stock, with the date each was last revised, to save the readers time. Some books say on their first page: “This is a reprint; the original is in the other shop.” Some have their pages glued shut until you shake them in a particular way (that’s JavaScript).

Search engines are the readers and the editors. SEO plumbing is everything you do so the readers can get in, find everything, understand what each page is and which version is the real one — and so the editors aren’t misled. It’s not the marketing side of SEO (keywords, content strategy). It’s the engineering side, and it’s yours.

Google’s pipeline (others are broadly similar):

  1. Crawl. Googlebot discovers URLs (from links, sitemaps, previous crawls) and fetches them, respecting robots.txt and managing crawl budget — how many URLs it’s willing to fetch from your site in a given time, based on your server’s health and the site’s perceived value.
  2. Render. For pages that return HTTP 200, Google queues them for rendering with an evergreen headless Chromium (the Web Rendering Service), executing JavaScript to see the final DOM. This used to be described as the “two waves” of indexing — HTML first, rendered content later — because rendering can be delayed. Today Google says most pages are rendered quickly, but rendering is still a separate, resource-limited step, and content that only appears after JS runs is at the mercy of it working: JS errors, blocked resources, slow APIs and things that require interaction (clicks, scrolls) mean content Google never sees.
  3. Index. Google analyses the rendered page — content, links, canonicals, structured data, language — decides whether to index it, and which URL is canonical among duplicates.
  4. Rank. When someone searches, hundreds of signals decide the order. Relevance and content quality dominate. Page experience (including Core Web Vitals) is one input among many.
Search engine pipelineDiscovery from links and sitemaps feeds a crawl queue governed by robots.txt and crawl budget. Fetched HTML with status 200 enters a render queue processed by headless Chromium running JavaScript. Rendered pages go to indexing, which applies canonicals, noindex and structured data. Indexed pages are ranked at query time. Red notes under each step show what can go wrong: disallowed, JS error, noindex or duplicate, low quality.Discoverlinks · sitemapsCrawlrobots.txt · budgetRenderheadless ChromiumIndexcanonical · noindexRankdisallowed / 5xxJS error / timeoutnoindex / duplicatethin / slow
Figure 22.1. Crawl → render → index → rank. Every step can silently drop your content: blocked from crawling, broken while rendering, excluded at indexing, or outranked.

robots.txt lives at the root of each host (https://lumen.example.in/robots.txt) and tells crawlers which paths they may crawl:

User-agent: *
Disallow: /admin/
Disallow: /search?
Allow: /
Sitemap: https://lumen.example.in/sitemap.xml

What it does not do:

  • It does not prevent indexing. A disallowed URL can still appear in search results (as a bare URL, without a snippet) if other sites link to it. To keep a page out of the index, it must be crawlable and say noindex (below). Disallowing it makes Google unable to see the noindex — a classic contradiction.
  • It is not security. It’s a public file that politely asks well-behaved bots to stay out. It lists exactly where your sensitive paths are. Protect private content with authentication.
  • It doesn’t apply across hosts. Each subdomain needs its own.
  • Blocking CSS/JS breaks rendering. If Googlebot can’t fetch your JS bundles, it renders an empty page. Don’t disallow /_next/ or /assets/.

The format was formally standardized in 2022 after decades as an informal convention.

A sitemap is a file listing the URLs you want search engines to know about. It helps discovery — especially for new sites, large sites, pages with few internal links, and recently changed content.

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<url>
<loc>https://lumen.example.in/essays/monsoon/</loc>
<lastmod>2026-11-02</lastmod>
</url>
<url>
<loc>https://lumen.example.in/essays/chai/</loc>
<lastmod>2026-10-18</lastmod>
</url>
</urlset>

The rules that matter:

  • Absolute, canonical URLs only. Exactly the URLs you want indexed: final, 200-status, not redirected, not noindex, matching your canonical tags (trailing slash and all).
  • Limits: a single sitemap file holds up to 50,000 URLs and 50 MB uncompressed. Bigger sites use a sitemap index that lists multiple sitemap files (by section: sitemap-essays.xml, sitemap-tags.xml).
  • lastmod honesty. Google uses lastmod if it’s consistently accurate, to decide what to recrawl. Set it to the date the page’s main content actually changed — not the build date, not “now” on every request. A sitemap where every URL claims to have changed today teaches Google to ignore your lastmod entirely.
  • changefreq and priority are ignored by Google. Skip them.
  • Generate it, don’t hand-write it. Next.js App Router supports a sitemap.ts route that returns entries from your data; Astro has an official sitemap integration (this book uses it); most frameworks have equivalents. Build-time generation for static sites; on-demand (and cached) for dynamic catalogues.
  • Tell search engines: reference it in robots.txt (Sitemap: line) and submit it in Google Search Console (and Bing Webmaster Tools), where you can see how many submitted URLs were indexed and why others weren’t. The old “ping” endpoint for notifying Google of sitemap changes has been retired; Google re-reads sitemaps it knows about.
Sitemap index structurerobots.txt points to a sitemap index, which lists three sitemaps: essays, tags and pages. Each sitemap lists URLs with lastmod dates. Search Console reads the index and reports indexed versus excluded counts.robots.txtsitemap-index.xmlsitemap-essays.xmlsitemap-tags.xmlsitemap-pages.xmlSearch Consoleindexed / excluded
Figure 22.2. Sitemap index and sitemaps. Each file lists canonical, indexable URLs with honest lastmod dates; search engines discover it via robots.txt or Search Console.

The same content often lives at several URLs: /essays/monsoon, /essays/monsoon/, /essays/monsoon?utm_source=whatsapp, http://…, www.…, a print version, an AMP version. Search engines must pick one to index (the canonical); you can tell them which:

<link rel="canonical" href="https://lumen.example.in/essays/monsoon/">

Rules: every indexable page has a self-referencing canonical with the absolute, preferred URL; canonical tags, internal links, sitemaps and redirects all agree on the same form (protocol, host, trailing slash); a canonical is a strong hint, not a command — if signals conflict, Google chooses for itself.

hreflang for multiple languages and regions

Section titled “hreflang for multiple languages and regions”

If Lumen publishes the monsoon essay in English and Hindi, hreflang tells search engines which URL serves which audience:

<link rel="alternate" hreflang="en-IN" href="https://lumen.example.in/en/essays/monsoon/">
<link rel="alternate" hreflang="hi-IN" href="https://lumen.example.in/hi/essays/monsoon/">
<link rel="alternate" hreflang="x-default" href="https://lumen.example.in/essays/monsoon/">

Rules: annotations must be reciprocal (each version lists all versions, including itself); use valid language (and optionally region) codes; x-default for the fallback/selector page; each URL is self-canonical. Get it wrong and the wrong language version ranks in the wrong country.

Page-level indexing instructions:

<meta name="robots" content="noindex, follow">
  • noindex — don’t include this page in search results (it must be crawlable for this to be seen).
  • nofollow — don’t follow links on this page.
  • nosnippet, max-snippet:N, max-image-preview:large — control how results look.

For non-HTML files (PDFs, images), or to apply at the server level, use the HTTP header X-Robots-Tag: noindex. Typical uses: internal search result pages, thin filter combinations, thank-you pages, staging and preview deployments (which brings us to the war story below).

Search engines take HTTP semantics seriously:

  • 200 — indexable content.
  • 301 / 308 — permanent move. Signals consolidate to the target. Use for changed URLs, domain migrations, http→https, enforcing trailing slashes.
  • 302 / 307 — temporary. The original URL stays indexed. Don’t use for permanent moves.
  • 404 / 410 — gone. The URL drops out of the index over time (410 marginally faster). Fine and normal for deleted content.
  • “Soft 404” — a page that returns 200 but says “not found” (common in SPAs that render a not-found component on a 200 shell). Search engines detect and treat these as errors; return a real 404.
  • 5xx / 429 — server trouble. Persistent errors reduce crawling and can drop pages from the index. A short outage is fine; days of 5xx are not.
  • Redirect chains (A→B→C→D) waste crawl budget and user round trips (Chapter 5). Redirect directly to the final URL.

Open Graph, Twitter cards — the share preview

Section titled “Open Graph, Twitter cards — the share preview”

When someone pastes Lumen’s link into WhatsApp, Slack, X or LinkedIn, those apps fetch the page and look for Open Graph (and Twitter card) meta tags to build a preview:

<meta property="og:title" content="Monsoon on Marine Drive">
<meta property="og:description" content="A photo essay on the first rain.">
<meta property="og:image" content="https://lumen.example.in/og/monsoon.jpg">
<meta property="og:url" content="https://lumen.example.in/essays/monsoon/">
<meta property="og:type" content="article">
<meta name="twitter:card" content="summary_large_image">

Not a ranking factor, but in a WhatsApp-first market like India, the share preview is your landing page. Use an absolute image URL, an image around 1200×630, and server-rendered tags (most link-preview bots don’t run JavaScript).

Structured data describes your page’s content in a machine-readable vocabulary (schema.org), usually as a JSON-LD script:

<script type="application/ld+json">
{ "@context": "https://schema.org", "@type": "Article",
"headline": "Monsoon on Marine Drive", "datePublished": "2026-07-14",
"author": { "@type": "Person", "name": "Lumen Editors" },
"image": ["https://lumen.example.in/og/monsoon.jpg"] }
</script>

Search engines use it to understand content and to show rich results (product prices and ratings, recipes, events, breadcrumbs, articles in Top Stories). Only certain types are eligible for rich results, eligibility rules change (some formerly common rich results have been restricted or retired), and markup must match the visible content. Validate with Google’s Rich Results Test. Structured data doesn’t directly boost ranking; it changes how your result looks, which affects clicks.

Core Web Vitals as a ranking signal — honestly

Section titled “Core Web Vitals as a ranking signal — honestly”

Google’s page experience signals, including Core Web Vitals, became part of ranking in 2021 (mobile) and 2022 (desktop). How much do they matter? Google’s consistent message: relevance and content quality dominate; page experience is a lighter signal that can matter when results are otherwise similar. Sites don’t leap from page three to the top by fixing CLS, and a slow page with uniquely relevant content still ranks.

So the honest pitch for performance work is not “it’ll fix our SEO.” It’s: performance helps users and conversion directly (Chapter 16), and it’s a modest positive for search on top. Use field data (CrUX via Search Console) because that’s what Google uses — not your Lighthouse score.

Client-rendered apps (Chapter 7) are where SEO plumbing most often breaks:

  • Empty initial HTML. Content exists only after JS runs; rendering failures or delays mean nothing gets indexed. Server-render or statically generate indexable pages.
  • Links that aren’t links. Navigation via onClick handlers on <div>s or buttons isn’t crawlable. Use real <a href> elements (your router’s Link component renders them).
  • Fragment URLs (/#/essays/monsoon): the fragment isn’t sent to servers and is generally not treated as a separate page. Use the History API with real paths.
  • Soft 404s: unknown routes render “Not found” with a 200 status. Return a real 404 from the server.
  • Per-route metadata: titles, descriptions, canonicals and OG tags must be in the server-rendered HTML per route, not set by client JS after load.
  • Content behind interaction: tabs, “load more” buttons and infinite scroll hide content from crawlers that don’t click or scroll. Provide paginated URLs with real links.
  • Blocked resources in robots.txt breaking rendering.

AI crawlers and robots.txt in the current era

Section titled “AI crawlers and robots.txt in the current era”

Alongside search crawlers, a new class of crawlers collects content for AI systems: some for training models, some for AI search and answers (fetching pages to cite in responses), and some acting on behalf of a user’s request in real time. Major AI companies publish separate user-agent tokens for these purposes (for example, OpenAI distinguishes its training crawler from its search crawler and its user-initiated fetcher; Anthropic, Perplexity, Apple, Common Crawl and others publish their own tokens; Google offers a separate Google-Extended token that controls use of content for its AI models without affecting Search indexing).

You can allow or disallow each in robots.txt:

User-agent: GPTBot
Disallow: /
User-agent: Google-Extended
Disallow: /

Considerations: blocking a training crawler doesn’t remove content already collected; blocking an AI search crawler can remove you from AI answers that would cite and link you; robots.txt is honoured voluntarily, and not every bot is well-behaved; CDNs now offer bot-management options, including blocking AI crawlers by default and charging them. Proposals like llms.txt (a curated markdown index for LLMs) exist but aren’t a standard that major search engines use. Decide deliberately, as a business choice, and revisit — this area is moving fast.

Crawler typesYour site in the centre. Arrows from four crawler types: search engine crawlers for indexing results; AI training crawlers collecting content for models; AI search crawlers fetching pages to cite in answers; user-initiated agents fetching a page on request. robots.txt tokens can allow or disallow each.your siteSearch crawlers → resultsAI training crawlersAI search (cite + link)user-initiated agents
Figure 22.3. Who's knocking. Search crawlers index for results; AI crawlers fetch for training, for AI search answers, or on a user's behalf — each controllable by its own robots.txt token.

When it goes wrong: the accidental noindex

Section titled “When it goes wrong: the accidental noindex”

This is a composite of a story that SEO consultants tell constantly, because it happens constantly.

A company redesigns its site. During development, the staging environment correctly carries <meta name="robots" content="noindex"> (and a Disallow: / in robots.txt) so search engines don’t index unfinished pages. Launch day: the new site goes live — and the staging configuration goes with it. The tag is in a shared layout template; the environment variable that should have switched it off was never set in production.

Nothing breaks. The site works perfectly for visitors. Over the next days and weeks, as Google recrawls, pages drop out of the index. Organic traffic slides 30%, 60%, 90%. Marketing notices a few weeks later and assumes it’s an algorithm update. Someone finally runs a URL inspection in Search Console: “Excluded by ‘noindex’ tag.” The fix takes one minute. Recovery takes weeks to months as pages are recrawled, and some lost rankings never fully return.

The variant with robots.txt is just as common: a Disallow: / carried from staging blocks all crawling; pages linger in results without snippets, then fade.

Lessons, as engineering:

  • Robots directives are configuration that differs by environment; test them like code. A CI check that fails the production build if the production HTML contains noindex or production robots.txt contains Disallow: /.
  • Protect staging with authentication, not just noindex (it also stops leaks of unreleased work).
  • Monitor: alert on sudden drops in indexed pages or organic clicks (Search Console data), and run a synthetic check that fetches the homepage and asserts no noindex (Chapter 23).

You can explain crawl, render, index and rank and where each fails; write robots.txt correctly and know its limits; produce honest sitemaps and sitemap indexes from your framework and submit them; set canonicals and reciprocal hreflang; use meta robots and X-Robots-Tag; choose correct status codes and avoid soft 404s; build share previews and structured data; put Core Web Vitals’ ranking weight in honest perspective; avoid SPA pitfalls; make a deliberate choice about AI crawlers; and prevent the accidental noindex.

1. Does Disallow in robots.txt keep a page out of search results?

No — it blocks crawling; the URL can still be indexed from links. Use crawlable noindex to exclude it.

2. What makes lastmod useful to Google?

Consistent accuracy — reflecting real content changes, not build times.

3. What’s a soft 404 and why is it bad?

A “not found” page returned with 200; search engines treat it as an error and it wastes crawl and confuses indexing.

4. Why must OG tags be server-rendered?

Most link-preview bots don’t execute JavaScript.

5. How much do Core Web Vitals affect ranking?

A modest signal; relevance and content quality dominate. The bigger payoff is user experience and conversion.

Warm-up 1. Spot the contradiction. A page is Disallowed in robots.txt and has noindex. What happens?

Solution

Google can’t crawl it, so it never sees noindex; the URL may still appear in results from external links. Allow crawling so noindex works.

Warm-up 2. Pick the status. An essay was renamed; a product is permanently discontinued with no replacement; a seasonal page will return next year.

Solution

301/308 to the new URL; 404 or 410; keep it live (200) with updated content, or 302 temporarily if it must redirect.

Warm-up 3. Canonical consistency. List four places the canonical form of a URL must agree.

Solution

Canonical tag, internal links, sitemap entries, redirects (and hreflang annotations).

Lab 4. Lumen’s plumbing. Add to Lumen: a generated sitemap.xml with real lastmod from essay data, robots.txt with the sitemap line, self-canonicals, per-essay titles/descriptions, OG/Twitter tags, and Article JSON-LD.

Expected outcome

View source on an essay shows all tags in server HTML; the Rich Results Test validates the JSON-LD; the sitemap lists only canonical URLs.

Lab 5. Search Console. Verify Lumen’s domain in Search Console (DNS TXT record — Chapter 5), submit the sitemap, and inspect one URL.

Expected outcome

The URL inspection shows crawl, render (screenshot of rendered HTML) and index status.

Lab 6. The noindex guard. Add a CI check that builds production and fails if any HTML contains a robots noindex (except an allow-list) or if robots.txt disallows /. Make preview deployments send X-Robots-Tag: noindex.

Expected outcome

Staging safety without risking production; the check fails if staging config leaks.

Lab 7. Share preview. Paste Lumen’s essay URL into WhatsApp (or a link-preview debugger). Fix whatever looks wrong.

Hint

Absolute image URL, correct size, server-rendered tags; caches in preview tools may need refreshing.

Staff 8. AI crawler policy. Write a one-page recommendation for your company’s stance on AI crawlers: training vs search vs user agents, business trade-offs, and review cadence.

Model points

Distinguish purposes; consider traffic from AI citations vs value of exclusivity; legal and licensing context; implementation in robots.txt plus CDN bot management; monitoring bot traffic; quarterly review.

Staff 9. Migration plan. Plan a URL structure migration for a 20,000-page site with minimal SEO loss.

Model outline

Full URL map; one-hop 301s; update internal links, canonicals, sitemaps, hreflang; keep old sitemap briefly to speed recrawl; monitor Search Console coverage and traffic by section; staged rollout; rollback plan.