Prerendering middleware

The problem

The bank's public site is a SPA built entirely from custom elements, and after the micro-frontend split every page component arrives through a runtime import() of a separately deployed remote. Search visibility is a hard requirement on that surface, and betting it on a crawler correctly executing a federated module graph — within its render budget, on every deploy — is not a bet worth placing.

The approach

Crawlers don't execute the app. A middleware in the site's own Express process detects bot traffic, renders the URL in headless Chromium, serializes the finished DOM, caches it and returns static HTML. Humans get the SPA on the same URL. The package is a few hundred lines with one real problem: the DOM the browser has built is not the DOM page.content() can see.

                        incoming request — one URL, one origin
                                      │
      ┌───────────────────────────────▼──────────────────────────────┐
      │  the product's own express app                               │
      │                                                              │
      │    prerender middleware                                      │
      │      bot user-agent?   ·   static asset?   ·   cached?        │
      │             │                                    │           │
      │       no ───┘  next()  ▸  SPA shell        bot,  │  miss     │
      └──────────────────────────────────────────────────┼───────────┘
                                                         │ CDP over ws
                                        ┌────────────────▼───────────────┐
                                        │  headless chromium pool        │
                                        │  separate service — scaled,    │
                                        │  restarted, upgraded on its    │
                                        │  own schedule                  │
                                        └────────────────┬───────────────┘
                                                         │
        goto ▸ networkidle0 ▸ strip scripts ▸ patch <base> ▸ serialize
                                                         │
                                        ┌────────────────▼───────────────┐
                                        │  cache — in-memory │ object    │
                                        │  store, keyed by full URL      │
                                        └────────────────┬───────────────┘
                                                         ▼
                                              finished HTML, real status

Key decisions

Force the polyfill, or serialize an empty page

This is the decision the project turns on. Nearly all rendered content lives inside shadow roots, and shadow roots aren't part of serialized markup — page.content() returns a structurally valid document of empty tags. Declarative Shadow DOM doesn't help, because the components render imperatively at runtime. What works is making the components render into the light DOM in the first place, by forcing the legacy polyfill before any app code runs:

if (cfg.useShadowDom) {
  await page.evaluateOnNewDocument('customElements.forcePolyfill = true');
  await page.evaluateOnNewDocument('ShadyDOM = {force: true}');
}

Both flags are read once, when elements are defined, so they must be installed with evaluateOnNewDocument — setting them after navigation is already too late. The output is readable HTML you can open in a browser and inspect, which is the debugging loop that scales. It is opt-in per site, so sites without shadow DOM don't pay for the polyfill.

Middleware in the app, not a rewrite at the edge

The usual setup is a separate prerender service plus edge rules that send bots to a different origin. Here, bots and humans hit the same URL on the same origin, served by the same deploy: no second hostname to get indexed by accident, no CDN rules to keep in sync with app routing, no new CORS or CSP surface. Dynamic rendering is only legitimate while crawlers get content equivalent to what people get, and one origin plus one codebase makes that structural rather than a promise.

A remote, pooled browser

puppeteer-core connects to a browser pool with puppeteer.connect({ browserWSEndpoint }) instead of launching Chromium in-process. The app image carries no browser, there are no sandbox flags to negotiate with the container runtime, and web replicas scale independently of browser memory.

One operational fact earns a comment in the compose file: Chrome allocates renderer shared memory from /dev/shm, Docker defaults it to 64MB, and under concurrent renders Chrome doesn't slow down — it crashes. Hence shm_size: 2g, with a note saying why, because it is exactly the setting someone deletes during a cleanup.

A render budget enforced by aborting requests

Navigation waits for networkidle0, the only readiness signal that works on every route without the app's cooperation. It also gives every third party a veto: a long-polling tag manager, a session-replay channel or a chat socket keeps the network busy and turns a two-second render into a full timeout. So every render intercepts requests and aborts anything matching a configurable pattern:

await page.setRequestInterception(true);

page.on('request', (req) => {
  Renderer.restrictRequest(req.url(), cfg)
    ? req.abort()
    : req.continue();
});

That is a latency control and a correctness one. Crawler traffic doesn't pollute analytics, ad pixels don't fire for bots, and a vendor outage can't become an SEO outage. Fonts and heavy media are dropped too, since they don't affect the serialized output.

Strip the scripts, keep the structured data

Left in place, the scripts would boot the SPA on the crawler's side and replace the markup that was just rendered. So they are removed before serialization, with a deliberately narrow selector:

document.querySelectorAll(
  'script:not([type]), script[type*="javascript"], script[type="module"], link[rel=import]'
).forEach(el => el.remove());

It removes classic scripts, ES modules and HTML imports, but leaves any <script> with a non-JavaScript type — including application/ld+json structured data, one of the most valuable things on the page for a search engine. A blanket querySelectorAll('script') would silently delete it, so the line carries a comment saying what it protects.

Never throw away a partial render

When navigation times out, Puppeteer rejects — but a 500 for a page that is 90% rendered is worse for a crawler than imperfect HTML. Each render captures the main-frame response as it arrives, whether or not goto ever resolves:

let response = null;
page.on('response', (rsp) => { if (!response) response = rsp; });

try {
  response = await page.goto(link, { timeout, waitUntil: 'networkidle0' });
} catch (e) {
  console.error(e);   // keep the captured response and serialize anyway
}

A timeout degrades to whatever had rendered, under the origin's real status code. Two smaller details sit next to it: an existing relative <base> is patched in place rather than duplicated, because only the first <base> in a document has any effect; and an upstream 304 is normalized to 200, since the crawler has no cached copy to reuse.

Two cache backends, and never cache a failure

Renders are expensive, and many crawlers request the same URL. One config key selects the backend: in-memory (a Map with per-entry TTL and a max-entry bound) for a single instance or local development, or an object store shared across replicas, with expiry handled by a bucket lifecycle rule instead of a sweeper in Node. Both key on the full URL, query string included.

What matters more is which responses are cacheable at all: an explicit status allowlist, 200 by default. Caching a 5xx would turn a five-minute upstream blip into a day of broken search results.

Built to be adopted by other sites

Static assets, API and file prefixes, and URLs carrying deep-link or campaign parameters short-circuit before a browser is ever contacted. Viewport, timeout, settle delay, abort pattern and cache backend are per-site config, so a new site adopts the package as a dependency plus a config block, not a fork.

When the browser pool's next major version changed how launch flags are passed — one url-encoded launch JSON object instead of bare query parameters — sites were on both versions at once. A single opt-in flag that rewrites the endpoint let each site migrate on its own schedule, and it was deleted once nothing was left on the old version.

Impact

• Improved search indexing across five public sites: crawlers receive complete, meaningful HTML for content that otherwise exists only inside shadow roots after JavaScript runs.
• Search visibility no longer depends on the client-side module graph, which let routes migrate into micro-frontends one at a time with no SEO regression.
• Each URL is rendered once per TTL and shared across replicas, so a full-site crawl isn't a load event.
• Third-party scripts are structurally excluded from renders: bot traffic never pollutes product metrics, and a vendor outage can't stall a render.
• The same HTML became the ingestion source for site search, so one tier pays for itself twice.