The public site of a consumer bank is a single-page app built entirely from custom elements, and after the split into micro-frontends every page component arrives through a runtime import() of a separately deployed remote. Search visibility is a hard product requirement on that surface, and betting it on a crawler executing a federated module graph — correctly, within its own render budget, on every deploy — is not a bet worth placing.

So crawlers don't execute it. A middleware inside the site's own Express process recognises bot traffic, renders the requested URL in a real headless Chromium, serialises the finished DOM, caches it, and returns static HTML. Humans get the SPA, untouched, on the same URL. The whole package is about 500 lines and one honest problem: the DOM the browser has finished building is not the DOM page.content() can see.

                        incoming request — one URL, one origin
                                      │
      ┌───────────────────────────────▼──────────────────────────────┐
      │  the product's own express app                               │
      │                                                              │
      │    prerender middleware                                      │
      │      bot user-agent?   ·   static asset?   ·   cached?        │
      │             │                                    │           │
      │       no ───┘  next()  ▸  SPA shell        bot,  │  miss     │
      └──────────────────────────────────────────────────┼───────────┘
                                                         │ CDP over ws
                                        ┌────────────────▼───────────────┐
                                        │  headless chromium pool        │
                                        │  separate service — scaled,    │
                                        │  restarted, upgraded on its    │
                                        │  own schedule                  │
                                        └────────────────┬───────────────┘
                                                         │
        goto ▸ networkidle0 ▸ strip scripts ▸ patch <base> ▸ serialize
                                                         │
                                        ┌────────────────▼───────────────┐
                                        │  cache — in-memory │ object    │
                                        │  store, keyed by full URL      │
                                        └────────────────┬───────────────┘
                                                         ▼
                                              finished HTML, real status

Decisions worth defending

Middleware in the application, not a rewrite rule at the edge. The common shape for this is a separate prerender service plus edge routing that sends bots to a different origin. This one lives in the app's own request pipeline. The bot and the human then hit the identical URL on the identical origin, served by the same deploy — which removes an entire category of problem: no divergent hostname to accidentally get indexed, no CDN rule to keep in sync with the app's routing, no CORS or CSP surface added, and nothing an auditor could reasonably read as cloaking. Dynamic rendering is only a legitimate technique while the content served to a crawler is equivalent to the content served to a person; keeping it inside one origin and one codebase makes that property structural rather than aspirational.

The browser is remote and pooled, never launched in-process. Using puppeteer-core with puppeteer.connect({ browserWSEndpoint }) rather than puppeteer.launch() means the app image carries no Chromium — no ~200MB download in the build, no sandbox flags to negotiate with the container runtime, and no coupling between web-tier replica count and browser memory. Chrome is a separately scaled service with its own concurrency limit and queue, upgraded on its own schedule. The cost of that separation is a network hop and one operational fact worth writing down: Chrome allocates shared memory for renderer processes out of /dev/shm, which Docker defaults to 64MB, and under concurrent renders it does not degrade — it crashes. The compose file pins shm_size: 2g with a comment saying exactly why, because it is precisely the kind of setting someone deletes during a cleanup.

Force the polyfill, or serialise an empty page. This is the decision the whole project turns on. The site's UI is custom elements, so essentially all rendered content lives inside shadow roots — and shadow roots are not part of the serialised markup. Native page.content() returns the light DOM: a document of empty tag names, structurally valid and worthless to a crawler. Declarative Shadow DOM shifts the problem rather than solving it, since the components render imperatively at runtime, not from server-authored templates. The approach that works is to make the components render into the light DOM in the first place, by turning on the legacy polyfill path before any application code evaluates:

if (cfg.useShadowDom) {
  await page.evaluateOnNewDocument('customElements.forcePolyfill = true');
  await page.evaluateOnNewDocument('ShadyDOM = {force: true}');
}

Both flags have to be installed via evaluateOnNewDocument — they are read once, at element-definition time, so setting them after navigation is too late by several hundred milliseconds. With the polyfill forced, component output lands in the light DOM, the serialiser sees the real page, and the emitted HTML is inspectable: you can open a prerendered response in a browser and read it, which is the debugging loop that scales here. It is opt-in per site, because a site not built on shadow DOM shouldn't pay the polyfill's cost.

A render budget enforced by aborting requests. Navigation waits for networkidle0 — zero in-flight requests — which is the only signal that generalises across every route without the application cooperating. It also hands a veto to every third party on the page. A tag manager that long-polls, a session-replay agent with an open channel, a chat widget's socket, a device-fingerprinting beacon: any one of them keeps the network permanently busy, and the render then costs the full timeout instead of two seconds. So request interception is on for every render, and anything matching a configurable pattern is aborted outright:

await page.setRequestInterception(true);

page.on('request', (req) => {
  Renderer.restrictRequest(req.url(), cfg)
    ? req.abort()
    : req.continue();
});

That is a latency control and, more importantly, a correctness control. Analytics shouldn't record crawler traffic as user sessions; ad pixels shouldn't fire for a bot; a third party having an outage shouldn't become an SEO outage. The same mechanism drops fonts and heavy media, which affect nothing in serialised output.

Strip the scripts, keep the structured data. Serialised HTML still contains every <script> tag from the original document, and returned unmodified the SPA boots on the crawler's side and replaces the markup the render just spent three seconds producing. So scripts come out before serialisation — but not indiscriminately:

document.querySelectorAll(
  'script:not([type]), script[type*="javascript"], script[type="module"], link[rel=import]'
).forEach(el => el.remove());

The selector is narrow on purpose. It removes classic scripts, ES modules and HTML imports — everything that would re-bootstrap the app — while leaving any <script> with a non-JavaScript type untouched. That exemption is the point: application/ld+json structured data is one of the most valuable things on the page for the consumer of this HTML, and a blanket querySelectorAll('script') would silently delete it. The line carries a comment saying what it protects, because the obvious simplification is wrong.

Patch the base href; don't add a second one. Prerendered HTML for a deep path contains relative URLs that were resolved by a browser sitting at that path. Served as-is, every relative link and asset resolves against whatever the consumer's base happens to be. The fix is a <base> tag pointing at the real origin and directory — with the wrinkle that only the first <base> in a document has any effect, so injecting one when the app already ships its own is a no-op that looks like a fix. The renderer patches an existing relative base in place, special-cases a bare "/", and injects only when there is none.

Never throw away a partial render. When navigation exceeds the timeout, Puppeteer rejects — and returning a 500 for a page that was 90% rendered is materially worse for a crawler than imperfect HTML. Every render therefore captures the main-frame response as it arrives, independently of whether goto ever resolves:

let response = null;
page.on('response', (rsp) => { if (!response) response = rsp; });

try {
  response = await page.goto(link, { timeout, waitUntil: 'networkidle0' });
} catch (e) {
  console.error(e);   // keep the captured response and serialize anyway
}

A timeout then degrades to “whatever had rendered, returned under the origin's real status code.” Status handling is deliberate in one more place: a 304 from the upstream — entirely possible on a repeat visit to the same origin with a warm browser cache — is normalised to 200, because a Not Modified carrying a full body against a request that had no cached copy is a response no client should have to reason about.

Two cache backends behind one interface, and never cache a failure. A render is expensive; the same URL is requested by many crawlers. Cache selection is one config key resolved to a class at construction, so the middleware has no idea which one it holds. In-memory is a Map with per-entry TTL and eviction at a max-entry bound — right for a single instance and for local development. The object-store backend is for the real deployment, where several replicas should share one render, and it expresses expiry as a bucket lifecycle rule rather than a sweeper in Node: the storage layer already does that job correctly. Both are keyed on the full incoming URL including query string. What matters more than either is which responses are eligible at all — an explicit status allowlist, 200 by default. Caching a 5xx would pin a transient upstream blip into search results for the entire TTL, converting a five-minute incident into a day-long one.

A browser-pool major upgrade shipped behind a boolean. The pool service changed its contract between major versions: Chrome launch flags could no longer be appended to the WebSocket endpoint as bare query parameters, and had to arrive as a single url-encoded launch JSON object. Sites were on both versions at once and would be for a while, so the fix could not be a breaking release of the middleware. It is one opt-in flag that rewrites the endpoint:

const NEW_VERSION_LAUNCH_ARGS = ['--ignore-certificate-errors', '--disable-web-security'];

function withLaunchArgs(endpoint, args) {
  const parsed = new URL(endpoint);
  parsed.searchParams.set('launch', JSON.stringify({ args }));
  return parsed.toString();
}

One middleware version speaks to both generations of the pool, each site flips the flag when its own infrastructure moves, and the flag deletes cleanly once nothing is left on the old version. Migration flags earn their keep when they make the rollout order somebody else's choice.

What the middleware declines to render is part of the contract. Requests are short-circuited before the browser is ever contacted: about forty static asset extensions plus the API, image and file prefixes never reach a render, and URLs carrying deep-link or campaign-attribution parameters bypass it entirely, because those are click-throughs whose value is the redirect, not the markup. Everything else about a render — viewport dimensions, timeout, an optional settle delay, the abort pattern, the cache backend — is per-site configuration, which is what makes the same package adoptable by a second site as a dependency and a config block rather than a fork.

Impact

• •Crawlers receive complete, structurally meaningful HTML for a site whose content otherwise exists only inside shadow roots after JavaScript execution.
• •Search visibility is decoupled from the client-side module graph — which is what allowed routes to migrate into independently deployed micro-frontends one at a time, with no SEO regression riding along with each move.
• •Render cost is paid once per URL per TTL and shared across replicas through the object-store cache, so a crawl of a large site is not a load event.
• •Packaged as a versioned middleware: several sites run it with different render dimensions, cache backends and third-party denylists.
• •Third-party scripts and analytics are structurally excluded from prerenders, so bot traffic never contaminates product metrics and a vendor outage cannot stall a render.
• •The same pre-rendered HTML became the ingestion source for site search, so one tier pays for itself twice.