Three public properties needed search: a consumer banking site, a research publication site, and a promotions catalogue behind a paginated JSON API. They shared no content store — each owned its own CMS and its own deploy pipeline — and each served every page in both Georgian and English. Two facts decided the architecture more than anything else. Georgian is agglutinative, so the word a user types is almost never the form on the page; and Georgian speakers routinely type Georgian words on a Latin keyboard, phonetically. A search box that matches tokens exactly returns nothing for most real queries in either case.

The service crawls each site through its own sitemap, extracts the readable text, builds a character trie per {channel, language} pair, and answers queries out of process memory. Ingestion is a scheduled job on worker threads; the HTTP layer only ever reads.

                       ┌─────────── indexing worker ────────────┐
  cron 09:00 ─────────▶│  per channel: sitemap XML → url list   │
  POST /refresh ──────▶│    ┌── chunk worker ─┐                 │
                       │    ├── chunk worker  ├─ HTML → text    │
                       │    └── chunk worker ─┘   (bot UA)      │
                       └───────────────┬────────────────────────┘
                                       │ partial postMessage
                                       │ { entries, cacheDelta }
                                       ▼
                     build a trie per { channel, language }
                                       │
                              merge — never wipe
                                       ▼
  GET /search ────────▶   ┌──────────────────────────────────┐
  x-channel, x-language   │ "CHANNEL-LANG" → search service  │
                          │ read-only · main thread · memory │
                          └──────────────────────────────────┘

Decisions worth defending

A trie in process memory, not a search cluster. The corpus is thousands of pages — tens of megabytes of extracted text, comfortably inside a Node heap. Against that, a managed search service is another deployment, another network hop on every query, and another failure domain. But cost and operability weren't what settled it: the language was. Off-the-shelf analyzers have no useful stemmer for Georgian, and a prefix trie is a crude stemmer — exactly the crude stemmer a language that inflects at the end of the word wants. Looking up სესხ reaches every document containing სესხი, სესხის, სესხები in one walk, with no linguistic rules to maintain. Lookup is O(length of query) and completely independent of corpus size.

Word-boundary insertion, document sets on the nodes. Every character is a node; a non-word character closes the current word and resets the cursor to the root, so each word of each indexed field is inserted from the root rather than the whole field being one long path. A node that terminates a word carries a Set of indices into that channel's data array, so a term appearing in both title and body costs one set entry, not two documents. Suggestions then come free — collect the subtree below the query node — and that collection walks an explicit stack rather than recursing, because trie depth here is the length of the longest word in the corpus and Georgian compounds are not short.

Query expansion at read time, not index time. Every query is walked in up to three forms: as typed, transliterated, and synonym-substituted. Transliteration is a greedy longest-match over a 33-entry table that tries three characters, then two, then one, so digraphs and the trigraph tch win before their single-character prefixes — which is what turns a Latin-typed sesxi into სესხი and finds the page. The direction is chosen by the index's own language, so the Georgian index transliterates Latin input and the English index does the reverse. Synonyms are stored as prefix pairs — credit ↔ loan, deposit ↔ savings — not whole words, which is the detail that makes them survive inflection: one pair covers every case ending the noun can take.

// three walks of the same trie, merged — not three indexes
const searchTerms = [queries];                      // as typed

const translated = this._getTranslation(queries);   // sesxi → სესხი
if (translated) searchTerms.push(translated);

const alt = queries.map(q => this._getSynonym(q) || q);
if (alt.some((term, i) => term !== queries[i])) searchTerms.push(alt);

Expanding at read time keeps the index a single representation of the corpus, so editing the synonym list is a config change rather than a reindex, and a bad synonym is reverted in a deploy instead of a crawl. The cost is bounded — at most three trie walks per query, each still proportional only to the length of what was typed, and a three-character minimum means the shortest queries never walk at all.

Ranking narrows the phrase rather than scoring the terms. A multi-word query is first matched whole; if nothing matches, the last term is dropped and the search retried, down to the first term. The score that falls out is matchedWords × 1000 + matchCount — deliberately lexicographic, so a longer phrase match always beats a shorter one no matter how often the shorter one occurs. Scoring lives in one class that receives the matched documents and the query terms, which keeps it replaceable behind a fixed interface.

Ingestion is a different program from query serving. The HTTP layer never crawls, never blocks on a network call and never writes to the index. That split isn't ceremony: crawling is slow and unreliable — timeouts, redirects, self-signed certificates inside the network, sites that are simply down — while queries have to be fast and always answerable. So indexing runs on a daily cron, once at startup, and on an explicit refresh endpoint, with a guard that makes a second concurrent run a no-op. /ready reports 503 until at least one index exists, so an orchestrator never routes traffic to a process whose tries are still empty, and it returns the job status — per-channel completion and errors — in the same payload.

Two levels of worker threads, with progress that lands incrementally. Indexing runs in a worker; that worker splits each channel's URL list into fixed chunks and hands them to chunk workers that pull from a shared cursor. A shared cursor rather than a fixed partition, because page fetch times differ by orders of magnitude — with a static split, most threads finish and idle while one straggles. More importantly, each finished channel posts its entries back to the main thread as a partial message and the index for that channel is built and installed immediately. The service warms channel by channel instead of holding everything back until the slowest crawl completes. Because that data crosses a worker boundary it has to survive structured cloning, so entries are serialized explicitly and non-cloneable fields are stripped rather than allowed to fail the whole run.

Merge, never replace. Storage is keyed CHANNEL-LANGUAGE, and a completed run swaps in only the keys it actually rebuilt.

const merged = Object.keys(next).length
    ? { ...SearchStorage.getAll(), ...next }  // fresh keys win, untouched keys survive
    : SearchStorage.getAll();

if (Object.keys(merged).length > 0) SearchStorage.replaceAll(merged);

Partial success is the normal outcome for a crawler, not the exception, and the naive version of this — rebuild everything, swap the whole map — means one unreachable site empties the search box for all of them. A channel that keeps failing keeps serving its previous index, which is the right behaviour for a marketing site and the reason job status is exposed on the readiness endpoint rather than only written to logs.

Indexed text and displayed text are different representations of the same field. Matching wants text with punctuation stripped, tags removed, whitespace collapsed, case folded. Result snippets built from that same normalized text read like a ransom note. So each field keeps its raw pre-sanitization form alongside the indexed one, and previews are cut from the raw text while matching happens on the normalized text. There are two snippet strategies behind one config flag: a character-scanning one, and a word-tokenizing one that keeps sentence punctuation, extends a configurable number of words past the match and can close on an ellipsis. The research channel publishes prose and needs the second; a product page with fifteen words of copy does not. That's a per-channel setting, because it is genuinely a property of the content.

The crawler identifies as a crawler. The sites being indexed are single-page apps whose served HTML contains no content until scripts run. The obvious fix — a headless browser in the ingestion path — means a browser to install, run and watch inside a container, for every page, every day. Instead the parser requests pages with a bot user-agent and receives the same server-side pre-rendered HTML that search engines get. Two things follow. Ingestion stays a plain HTTP fetch with no runtime to babysit; and the site index is, by construction, built from the same HTML a search engine indexes — so a page invisible to search engines is also invisible to site search, which is the correct coupling rather than a coincidence.

Extraction opt-out lives in the markup. Navigation, footers, cookie banners and date pickers are text, and left alone they pollute every result with the same boilerplate. There's a default selector list, per-channel additions in config, and a [data-search-ignore] attribute any element can carry. The attribute matters most: a central selector list maintained in this repo describes components owned by other repos, and it starts rotting the day a component is renamed. The attribute lets the team that owns a component exclude it at the source, where the rename happens.

Re-fetching is decided by the sitemap, not by a heuristic. Crawling every page daily wastes hours of wall clock and hammers the sites being indexed, and the answer to “did this change” is already published: sitemaps carry lastmod. Parsed content is cached per URL and reused when the sitemap's lastmod is unchanged, so a normal daily run refetches only what moved. The cache carries a version number, and bumping it invalidates the whole thing in one line — the lever for the day the extraction rules change and every cached page is subtly stale.

Making it extensible by other people

Adding a channel is deliberately not an architecture conversation: add an entry to the config file, implement a <CHANNEL>Parser method, and for sitemap-based sites that method is one line delegating to the shared parser. Everything else — languages, indexed fields, ignored selectors, preview shape, whether the original API item is echoed back in results — is data in the config entry, documented field by field. The one channel that isn't sitemap-based reads a paginated JSON API instead, and it needed no change to the indexing or search core to do it, which is the only real test of whether that boundary was in the right place.

The second thing that made it maintainable by others was making the index inspectable. A paginated endpoint dumps exactly what the parser extracted for a given channel and language, page by page. “Why doesn't my page come up in search” is otherwise a log-reading exercise for whoever owns the service; with the extracted text visible, it's usually self-evident within a minute — the content sits inside a component the selector list strips, or the page returned a shell because pre-rendering failed for it. That endpoint turned a recurring class of bug report into something the reporter could answer.

Impact

• •Site search across three properties and two languages runs as one container with no search infrastructure behind it — nothing to provision, tune, or pay for per query.
• •Query cost is proportional to the length of what the user typed, not to the size of the corpus, and never involves a network call.
• •Georgian typed in Georgian and Georgian typed phonetically in Latin resolve to the same documents, as do inflected forms and the credit/loan class of domain synonyms.
• •A channel that fails to crawl degrades to its previous index alone; the others keep serving, by construction rather than by luck.
• •A normal daily run refetches only the pages the sitemap reports as changed, and the service warms channel by channel rather than all at once.
• •New channels arrive as a config entry plus one parser method, added by engineers who never touched the indexing or search core.