LLM-assisted search API

The problem

The site's keyword index worked when people typed a product name and returned nothing when they didn't. Real queries are phrases (what do I need for a car loan) and typos, in two languages.

The approach

Rather than replace the index, an LLM rewrites the query into indexed terms before it reaches the index, and a grounded answer card sits above the results. The card is retrieval-augmented generation (RAG) with the existing keyword index as the retriever: the index retrieves the pages and the model only summarizes them. Search-as-you-type autocomplete runs on the trie-based search service.

query ───▶ rewrite (LLM) ───▶ keyword index ───▶ ranked results     blocking · fast path
            │                                       │
            ├─ single token → spelling correction   │
            └─ phrase       → keyword extraction    │
                                                    ▼
                            top-2 result pages ──▶ LLM ──▶ answer card     cached · streamed

The query rewrite

Two prompts, not one clever one

Fixing a typo and extracting intent from a sentence fail in opposite directions: a spelling prompt given a question invents a product, and an extraction prompt given one misspelled word “corrects” it into a different product. A deterministic check on the query's shape — not a model call — picks the prompt, so each one has a single job and a checkable output:

// one dispatcher, two single-purpose prompts, one fallback contract
const rewrite = async (query, lang) => {
  const task = query.trim().includes(' ') ? EXTRACTION : SPELLING;
  try {
    const out = await callModel(promptFor(task, query, lang), MODEL_OPTS);
    return out?.trim() || query;   // empty or unusable → raw query
  } catch (err) {
    log(err);
    return query;                  // model failure → pre-AI search, never an error
  }
};

Fail back to the old search, never to an error

Every model call falls back to the user's raw query, so a provider outage silently returns the pre-AI keyword search, and a failed summary returns null so the card simply doesn't render. The whole layer is gated by a config key plus a request parameter and can be turned off without a deploy.

Constrain the model by construction

Decoding is deterministic, and output is capped at sixteen tokens for a rewrite and a few hundred for a summary. The cap bounds cost and truncates runaway output before it reaches the index; the caller still normalizes the result and falls back when it is empty. Every call has separate connect and request timeouts, so a third party on the critical path never gets an unbounded wait.

The cheapest model tier, on purpose

The rewrite is a classification-shaped task with a one-term answer, and the summary is extractive. Neither benefits from a frontier model, and both suffer from its latency and cost. Because outputs are small and bounded, the tier is a config line per call site rather than an architectural commitment. Calls go through the Vercel AI SDK, which makes the provider a config line too and gives streaming, timeouts and abort signals one interface instead of one per vendor.

The answer card

Two endpoints, so the slow call can't slow the list

The rewrite is cheap and on the critical path; summarization has to fetch and read pages. So results come from a blocking GET, and the summary is a separate POST fired after results paint, streaming into a skeleton placeholder so nothing reflows. The first sentence appears while the rest is still generating, and summary latency reads as time to first token rather than time to finish.

Grounded in pages the index already returned

On a bank site, an invented answer is the failure that matters. The card is generated only from the top two ranked pages, fetched server-side at request time. Those pages belong to a SPA, so the fetch reuses the pre-render path the site serves to search bots — in parallel, under a hard timeout — and only the document body reaches the prompt. The card sits directly above the results it cites, and provider safety thresholds are set explicitly rather than left at defaults.

Schema-validated streaming, not parsed free text

The card is generated with the AI SDK's streamObject against a schema, so there is no fence-stripping or hand-written JSON parsing to get wrong. Partial fields render as they stream, but the finished object is validated: a response that fails the schema leaves no card on the page and is never cached.

Models also drift to English on non-English input, so the target language is pinned in the prompt and included in the cache key. The two locales can never serve each other's text.

Cost

Cache on the rewritten keyword, not the raw query

This is the highest-leverage line in the feature. Hundreds of phrasings and misspellings collapse onto one (language, keyword) cache entry, so the long tail hits cache instead of paying for page fetches and generation. The TTL follows the content-release cadence — a staleness budget, not a cost knob. Only the summary is cached, never the ranked results, so the list is always current and the live page is one click below the card.

Responses also report whether the rewrite ran and whether the summary came from cache, so AI-path share and hit rate are readable from ordinary request logs.

Prompt caching for the misses

Every cache miss still sends a full prompt, so prompts are laid out static-first: instructions, output contract and language rules at the top, the query and page text at the end. The long identical prefix is served from the provider's prompt cache. The two caches work at different layers — one skips the model call entirely, the other makes the calls that do happen cheaper.

The results page

Independent sources fail independently

Results combine more than one upstream index plus an editorial exclusion list. They are queried concurrently, and the non-essential ones resolve to an empty set on failure, so one source being down narrows the results instead of blanking the page. Exactly one upstream is a hard dependency — a deliberate choice about which failure users are allowed to see.

The same page can arrive from several sources, with titles wrapped in query-dependent highlight markup, so a naive comparison never matches. Deduplication uses the URL with case and trailing slash normalized plus the title with markup stripped, while the highlighted title is kept for display.

No stale responses

A search-as-you-type box with a slow second request is a race generator. Input is debounced by half a second behind a three-character minimum, each request gets a monotonic id, and superseded requests are aborted — for results and summary independently:

this._requestId = (this._requestId || 0) + 1;
const requestId = this._requestId;
this._pending?.cancel();                      // abort the in-flight request

this._pending = this.apiCall(searchUrl(term));
this._pending.then((res) => {
  if (requestId !== this._requestId) return;  // superseded → drop the response
  render(res);
});

The same guard is the cheapest cost control in the feature: most keystrokes never become a request, so the model only sees queries a user paused on.

One component, two consumers

Fetching, debounce, race handling and state live in a base Lit element shared through a Module Federation remote; the header modal and the full results page extend it. Summarization hangs off a single _onProductsLoaded hook the subclass may implement, so the lightweight modal never pays for it and shared code has no if (isModal) branches.

Impact

• Clicks on search results rose by 50%.
• Questions, phrasings and misspellings resolve to indexed terms instead of an empty results page.
• The keyword search is the floor: no query returns worse results than before, and the layer can be switched off from config.
• Cost per search is bounded by construction — capped rewrites, a keyword-level response cache, and prompt caching for everything else.
• Every claim on the answer card comes from a ranked page one click away.