A consumer bank's public site had a keyword search index that worked well when people typed a product name and returned nothing when they didn't. Real queries aren't product names — they're phrases (what do I need for a car loan) and typos, in two languages. Rather than replace the index, I put an LLM query-understanding layer in front of it and added a grounded answer card above the results.

query ──▶ rewrite (LLM) ──▶ keyword index ──▶ ranked results     blocking · fast path
            │                                       │
            ├─ single token → spelling correction   │
            └─ phrase       → keyword extraction    │
                                                    ▼
                            top-N result pages ──▶ LLM ──▶ answer card     cached, non-blocking

Decisions worth defending

Two prompts, not one clever one. Correcting a misspelled term and reading intent out of a sentence are different tasks that fail in different directions — a spelling prompt handed a question invents a product, an extraction prompt handed one typo'd word “corrects” it into a different product. A cheap deterministic check on the shape of the query — not a model call — routes it to one or the other, so each prompt has a single job and a checkable output:

// one dispatcher, two single-purpose prompts, one fallback contract
const rewrite = async (query, lang) => {
  const task = isSingleToken(query) ? SPELLING : EXTRACTION;
  try {
    const out = await callModel(promptFor(task, query, lang), MODEL_OPTS);
    return out?.trim() || query;   // empty or unusable ⇒ raw query
  } catch (err) {
    log(err);
    return query;                  // model failure ⇒ pre-AI search, never an error
  }
};

Two endpoints, not one. The rewrite is cheap and on the critical path; summarization needs to fetch and read the result pages, so it can't be. Results are a blocking GET; the summary is a second POST fired after results paint, into a skeleton placeholder so nothing reflows. The expensive call can't regress the latency of the list.

Failure degrades to the old behavior, never to an error. Every model call is wrapped so its fallback value is the user's raw query — a provider outage silently returns the pre-AI keyword search. The summary returns null and the card simply doesn't render. The whole layer is gated on a flag plus a request param, so it can be switched off without a deploy.

The cheapest model tier, on purpose. The rewrite is a classification-shaped task whose answer is a single term, and the summary is extractive — neither rewards a frontier model, and both are sensitive to the latency and cost one adds. Picking the smallest tier was a decision the design then had to earn: because the output contract is a bounded token count and a small fixed object, the tier is a config line rather than an architectural commitment, and it can be revisited per call site if quality ever demands it.

Cache on the rewritten keyword, not the raw query. This is the highest-leverage line in the feature: hundreds of phrasings and misspellings collapse onto one (language, keyword) cache entry, so the long tail hits cache instead of paying for page fetches and generation. The TTL is set against the content-release cadence rather than picked for cost, which makes it a staleness budget — worth naming on a bank surface: a page edited after its card was generated stays described by that card until the entry expires. Two things bound it. Only the generated summary is cached — the ranked results are not, so the list stays current even when the card is behind it — and the card renders directly above those results and cites the pages it read, so the live page is always one click beneath the summary of it. Time-based expiry is still the weaker mechanism, and it's the second reason publish-time extraction is the change I'd make first: it turns invalidation into an event instead of a timeout. Responses also carry whether the rewrite ran and whether the summary came from cache, so AI-path share and hit rate are readable from ordinary request logs instead of needing their own instrumentation.

Constrain the model by construction, not by asking. Deterministic decoding, and an output cap sized for a term rather than a sentence: the cap bounds cost and truncates runaway output before it reaches the index. It doesn't make a sentence impossible — a max-tokens setting cuts the response off rather than constraining its shape — so the caller still normalizes what comes back and falls through to the raw query when the result is empty or truncated to nothing. Separate connect and request timeouts on every call: a third party on the critical path never gets an unbounded await. JSON responses get fence-stripped before parsing (models wrap output in markdown code fences regardless of instructions) and are validated before reaching the client.

Reading the site the way a crawler does. The pages being summarized belong to a single-page app, so fetching one server-side returns an empty shell — the text worth summarizing doesn't exist until a client renders it. The fetch therefore reuses the same pre-render path the site already exposes to search-engine bots, in parallel across the selected pages and under a hard timeout. Only the document body is extracted before the HTML reaches the prompt, which drops the scripts and metadata that would otherwise dominate the token count.

Grounding over free-form answering. On a bank surface, an invented answer is the failure mode that matters. The summary is generated only from the HTML of the top-ranked pages the index already returned, fetched server-side at generation time, and it renders only if the model produced both fields of the expected contract. It sits directly above the real results, so the source is one click away. Provider safety thresholds are set explicitly rather than left at defaults.

Partial degradation across independent sources. A result page draws on more than one upstream system, and they don't share a failure domain. They're issued concurrently rather than awaited in sequence, and the non-essential ones resolve to an empty set on failure instead of rejecting: one source being down narrows the results, it doesn't produce a blank page. Exactly one upstream is a hard dependency, which is a deliberate choice about which failure the user should be allowed to see.

Dedupe on normalized identity, not on rendered text. The same page legitimately arrives from more than one source, and the index returns titles with the matched span wrapped in highlight markup — so the naive comparison never matches, because the markup itself varies with the query. Identity is the URL with trailing slash and case normalized, plus the title with highlight markup stripped; the visible, highlighted title is left untouched for rendering. Small function, but it's the difference between a clean result list and the same product listed twice.

Bilingual output is enforced, not hoped for. Models drift to English on non-English input. Target language is pinned in the prompt and is part of the cache key, so the two locales can never serve each other's text.

Client-side race correctness. A search-as-you-type box with a slow second request is a race generator. Input is debounced, each request carries a monotonic id, and in-flight requests are aborted on supersede — for both the results and the summary independently. A stale response can't overwrite a newer one, which is the bug users report as “it showed me the wrong results for a second.”

this._requestId = (this._requestId || 0) + 1;
const requestId = this._requestId;
this._pending?.cancel();                      // abort the in-flight request

this._pending = this.apiCall(searchUrl(term));
this._pending.then((res) => {
  if (requestId !== this._requestId) return;  // superseded — drop the response
  render(res);
});

The same guard doubles as the cheapest cost control in the feature: a minimum query length and the debounce window mean the overwhelming majority of keystrokes never become a request at all, so the model only ever sees queries a user paused on.

One component, two consumers. The search logic (fetch, debounce, race handling, state) lives in a base Lit element shared through a Module Federation remote; the header modal and the full results page extend it. Summarization hangs off a single _onProductsLoaded hook the subclass may implement — so the lightweight modal never pays for it, and there's no if (isModal) branching in shared code.

Alignment outside engineering

I built this one end to end — both endpoints, both prompts, the caching layer and the client component — but the part users actually see was not mine to decide alone. An answer card on a bank's site is a new kind of surface: it reads as authoritative, it sits above the real results, and it is generated. What it should look like, how confident it is allowed to appear, and how it relates visually to the results below it went through design. The disclosure — that the answer is AI-generated, and how to indicate that Gemini is the model behind it — went through copyright and legal review, which set the wording and the attribution rather than leaving them to me. Both are constraints that are cheap while they are shaping a design and expensive once they arrive after it, which is why I pulled them in before the card had a final shape.

Impact

• •Questions, phrasings and misspellings resolve to indexed terms instead of an empty-state page.
• •The pre-AI keyword path is the floor, not the fallback of last resort: no query can return worse results than it did before, and the layer can be turned off from config.
• •Cost per search is bounded by construction — a capped-length rewrite and a cache keyed on the normalized keyword, so repeat traffic never reaches the provider.
• •Popular queries serve their answer card from cache, so the expensive path is paid once per keyword per TTL window rather than once per search.

What I'd change

Fetching page HTML on every cache miss is the slowest step by a wide margin, and extracting page content at publish time instead of query time is the fix. I'd sequence it first because it pays twice: it removes the fetch from the request path, and it turns the summarized text into a stored artifact that can be normalized and bounded once rather than parsed fresh on every miss. Reading first-party, editorially controlled pages at query time is what makes the current shape reasonable — a publish-time extract is the version that stays correct regardless of who owns a page later. I'd move JSON output onto the provider's structured-output feature in the same pass and drop the fence-stripping with it, and stream the card rather than landing it all at once.