Several public marketing and product sites needed search: a consumer banking site, a research publication site, and a promotions catalogue behind a paginated JSON API. They shared no content store — each owned its own CMS and its own deploy pipeline — and they served every page in both Georgian and English. Two facts decided the architecture more than anything else. Georgian is agglutinative, so the word a user types is almost never the form on the page; and Georgian speakers routinely type Georgian words on a Latin keyboard, phonetically. A search box that matches tokens exactly returns nothing for most real queries in either case.

What I built is a small Node service that crawls each site through its own sitemap, extracts the readable text, builds a character trie per {channel, language} pair, and answers queries out of process memory. Ingestion is a scheduled job on worker threads; the HTTP layer only ever reads.

                       ┌────────── indexing worker ───────────────┐
  cron 09:00 ─────────▶│ per channel:  sitemap XML → url list     │
  POST /refresh ──────▶│     ├── chunk worker ─┐                  │
                       │     ├── chunk worker  ├─ HTML → text     │
                       │     └── chunk worker ─┘   (bot UA)       │
                       └───────────────┬──────────────────────────┘
                                       │ partial postMessage
                                       │ { entries, cacheDelta }
                                       ▼
                     build a trie per { channel, language }
                                       │
                             merge — never wipe
                                       ▼
  GET /search ────────▶   ┌──────────────────────────────────┐
  x-channel, x-language   │ "CHANNEL-LANG" → search service  │
                          │ read-only · main thread · memory │
                          └──────────────────────────────────┘

Decisions worth defending

A trie in process memory, not a search cluster. The corpus is thousands of pages — tens of megabytes of extracted text, comfortably inside a Node heap. Against that, a managed search service is another deployment, another network hop on every query, and another failure domain to hold open at 3am. But cost and operability weren't what settled it: the language was. Off-the-shelf analyzers have no useful stemmer for Georgian, and a prefix trie is a crude stemmer — exactly the crude stemmer a language that inflects at the end of the word wants. Looking up სესხ reaches every document containing სესხი, სესხის, სესხები in one walk, with no linguistic rules to maintain. Lookup is O(length of query) and completely independent of corpus size. What I gave up is real: no fuzzy matching, no ranking model worth the name, and an index that lives and dies with the process.

Word-boundary insertion, document sets on the nodes. Every character is a node; a non-word character closes the current word and resets the cursor to the root, so each word of each indexed field is inserted from the root rather than the whole field being one long path. A node that terminates a word carries a Set of indices into that channel's data array, so a term appearing in both title and body costs one set entry, not two documents. Suggestions then come free — collect the subtree below the query node — and that collection walks an explicit stack rather than recursing, because trie depth here is the length of the longest word in the corpus and Georgian compounds are not short.

Query expansion at read time, not index time. Every query is walked in up to three forms: as typed, transliterated, and synonym-substituted. Transliteration is a greedy longest-match over a digraph table (sh, ch, dz, tch before their single-character prefixes), which is what turns a Latin-typed sesxi into სესხი and finds the page. Synonyms are stored as prefix pairs — კრედიტ → სესხ, credit → loan — not whole words, which is the detail that makes them survive inflection: one pair covers every case ending the noun can take.

// three walks of the same trie, merged — not three indexes
const searchTerms = [queries];                      // as typed

const translated = this._getTranslation(queries);   // sesxi → სესხი
if (translated) searchTerms.push(translated);

const alt = queries.map(q => this._getSynonym(q) || q);
if (alt.some((term, i) => term !== queries[i])) searchTerms.push(alt);

Expanding at read time keeps the index a single representation of the corpus, so editing the synonym list is a config change rather than a reindex, and a bad synonym is reverted in a deploy instead of a crawl. The cost is bounded — at most three trie walks per query, each still proportional only to the length of what was typed.

Ingestion is a different program from query serving. The HTTP layer never crawls, never blocks on a network call and never writes to the index. That split isn't ceremony: crawling is slow and unreliable — timeouts, redirects, self-signed certificates inside the network, sites that are simply down — while queries have to be fast and always answerable. So indexing runs on a daily cron, once at startup, and on an explicit refresh endpoint, and /ready reports 503 until at least one index exists, so an orchestrator never routes traffic to a process whose tries are still empty.

Two levels of worker threads, with progress that lands incrementally. Indexing runs in a worker; that worker spawns chunk workers that pull URL batches from a shared cursor. A shared cursor rather than a fixed partition, because page fetch times differ by orders of magnitude — with a static split, most threads finish and idle while one straggles on a slow channel. More importantly, each finished channel posts its entries back to the main thread as a partial message and the index for that channel is built and installed immediately. The service warms channel by channel instead of holding everything back until the slowest crawl completes.

Merge, never replace. Storage is keyed CHANNEL-LANGUAGE, and a completed run swaps in only the keys it actually rebuilt.

const merged = Object.keys(next).length
    ? { ...SearchStorage.getAll(), ...next }  // fresh keys win, untouched keys survive
    : SearchStorage.getAll();

if (Object.keys(merged).length > 0) SearchStorage.replaceAll(merged);

Partial success is the normal outcome for a crawler, not the exception, and the naive version of this — rebuild everything, swap the whole map — means one unreachable site empties the search box for all of them. The trade I accepted is that a channel which keeps failing keeps serving a stale index indefinitely; that's the right call for a marketing site, and it's the reason the job status (per-endpoint completion and errors) is exposed on the readiness endpoint rather than only written to logs.

Indexed text and displayed text are different representations of the same field. Matching wants text with punctuation stripped, tags removed, whitespace collapsed, case folded. Result snippets built from that same normalized text read like a ransom note. So each field keeps its raw pre-sanitization form alongside the indexed one, and previews are cut from the raw text while matching happens on the normalized text. There are two snippet strategies behind one config flag: a character-scanning fallback, and a word-tokenizing one that keeps sentence punctuation and can extend further past the match. The research channel publishes prose and needs the second; a product page with fifteen words of copy does not. That's a per-channel setting, not a global one, because it is genuinely a property of the content.

The crawler identifies as a crawler. The sites being indexed are single-page apps whose served HTML contains no content until scripts run. The obvious fix — a headless browser in the ingestion path — means a browser to install, run and watch crash-loop inside a container, for every page, every day. Instead the parser requests pages with a bot user-agent and receives the same server-side pre-rendered HTML that search engines get. Two things follow. Ingestion stays a plain HTTP fetch with no runtime to babysit; and the site index is, by construction, built from the same HTML Google indexes — so a page invisible to search engines is also invisible to site search, which is the correct coupling rather than a coincidence. It also means the pre-render tier already justified by SEO pays for itself twice.

Extraction opt-out lives in the markup. Navigation, footers, cookie banners and date pickers are text, and left alone they pollute every result with the same boilerplate. There's a default selector list, per-channel additions in config, and a [data-search-ignore] attribute any element can carry. The attribute matters most: a central selector list maintained in this repo describes components owned by other repos, and it starts rotting the day a component is renamed. The attribute lets the team that owns a component exclude it at the source, where the rename happens.

Re-fetching is decided by the sitemap, not by a heuristic. Crawling every page daily wastes hours of wall clock and hammers the sites being indexed, and the answer to “did this change” is already published: sitemaps carry lastmod. Parsed content is cached per URL and reused when the sitemap's lastmod is unchanged, so a normal daily run refetches only what moved. The cache is versioned, and a version bump invalidates the whole thing — the escape hatch for the day the extraction rules change and every cached page is subtly wrong.

Making it someone else's to extend

Adding a channel is deliberately not an architecture conversation: add an entry to the config file, implement a <CHANNEL>Parser method, and for sitemap-based sites that method is one line delegating to the shared parser. Everything else — languages, indexed fields, ignored selectors, preview shape, whether the original API item is echoed back in results — is data in the config entry, documented field by field in the README. Other engineers added channels and tuned extraction rules without touching the indexing or search core, which is the only real test of whether that boundary was in the right place.

The second thing that made it maintainable by others was making the index inspectable. A paginated endpoint dumps exactly what the parser extracted for a given channel and language, page by page. “Why doesn't my page come up in search” is otherwise a log-reading exercise for whoever owns the service; with the extracted text visible, it's usually self-evident within a minute — the content sits inside a component the selector list already strips, or the page returned a shell because pre-rendering failed for it. That endpoint turned a recurring class of bug report into something the person reporting it could answer.

Impact

• •Site search across three properties and two languages runs as one container with no search infrastructure behind it — nothing to provision, tune, or pay for per query.
• •Query cost is proportional to the length of what the user typed, not to the size of the corpus, and never involves a network call.
• •Georgian typed in Georgian and Georgian typed phonetically in Latin resolve to the same documents, as do inflected forms and the credit/loan class of domain synonyms.
• •A channel that fails to crawl degrades to a stale index for that channel alone; the others keep serving, by construction rather than by luck.
• •A normal daily run refetches only the pages the sitemap reports as changed.

What I'd change

Ranking is the weakest part and I'd fix it first. The score is matchedWords × 1000 + matchCount — deliberately lexicographic, so a longer phrase match always beats a shorter one — but underneath that there is no length normalization and no field weighting, so a long page mentioning a term five times outranks a short page whose title is the term. Scoring is already isolated in one class that receives the matched documents and the query terms, so a BM25-style score with a title boost is a contained change behind the same interface. That containment was intentional; not having spent it yet is the omission.

There's no typo tolerance, and a trie is unforgiving about it: a prefix is either present or absent, so one wrong character in the middle of a word returns an empty result rather than something close. The cheap version of the fix is well understood — on zero results, walk the trie with a bounded edit distance instead of exact character matching, and offer the nearest few terms as suggestions. It costs one extra traversal on exactly the queries that currently fail, which is the right place to spend it.

The failure mode I'd design away is the cold start. The incremental cache lives in process memory, so a restart — a deploy, a node eviction, an OOM — discards it and the next run re-crawls every page of every channel from scratch. The daily window absorbs it and nothing user-visible breaks, but it is a self-inflicted burst of traffic against the very sites this service depends on, triggered by something as routine as a deploy. The cache is already versioned and serialized at the worker boundary, so persisting it to disk or a shared store is a small change with a disproportionate payoff.

Last, channel dispatch resolves by string convention: the config names a channel, and the parser method is looked up by name at run time. A config entry with no matching parser logs an error and yields an empty index — a silent hole in search results that surfaces as a support ticket days later. An explicit registry, validated when the process boots, turns that into a startup failure, which is when you actually want to hear about it.