Search indexing basics
An index from words to documents, so "find me things containing this" is not a full scan.
What problem does a search index solve?
Databases match values. Search matches meaning, roughly. A LIKE '%shoe%' query cannot use a normal index, cannot handle typos or word endings, and has no idea which result is better than another.
A search engine builds an inverted index: for each word, the list of documents containing it. So a query becomes a few list intersections, and the results come back ranked.
What a search index costs you
Every concept in here states a cost. A technique with no price is a technique you have not understood yet — and naming the cost out loud is most of what separates a senior answer from a junior one.
When to reach for a search index, and when not to
Use it when
- Full-text search over a meaningful amount of text.
- Faceted search: filter by brand, price, rating, with counts.
- Typeahead and autocomplete.
- Searching logs and events, where the volume makes a database impractical.
- Ranking by relevance rather than by a column.
Do not, when
- Exact lookups by id or by a known field. Your database does that better.
- Small datasets. Most relational databases have decent built-in full-text search, and one system beats two.
- You need transactional guarantees. A search index is derived data and should never be the source of truth.
How a search index actually works
Analysis is the step that decides quality, and it gets the least attention. Text is split into words, lowercased, sometimes stripped of very common words, and cut back to a root form, so "running" and "runs" both become "run". Both the document and the query go through the same steps, which is why they match. Change the analyser and you must rebuild the index, because the old one holds words in the old form.
Ranking, at the level worth explaining. A word that appears often in this document is a stronger signal. A word that appears in almost every document is a weaker one. That combination is TF-IDF. BM25 is the refined version, and the default scoring in Lucene-based engines. On top of the text score you add business signals: popularity, how recent it is, whether it is in stock, the margin. In product search, much of the tuning effort goes into that second part.
For typeahead specifically, the general engine is often the wrong tool. Matching a prefix against a few million terms is better served by a purpose-built structure: a trie with the best completions worked out in advance at each node, held in memory. That gives single-digit millisecond responses. Prepare the answers rather than searching for them, because the user types another character in 200 ms and every keystroke is a query.
Keeping it in step: drive indexing from the database log or from an outbox, rather than writing to both places from the application. Index whole documents, so re-indexing a record is safe to repeat. And plan for a full rebuild, because you will change the mapping. The standard approach is to build a new index alongside the old one, then switch an alias over in one step, so there is never a moment with no index.
The shape at scale: search indexes are split into shards, and each shard is copied. A query goes to every shard, each returns its top N, and a coordinator merges them. So query cost grows with the number of shards, and having far more shards than you need makes every query slower. Deep pagination is expensive for the same reason. Page 500 means each shard has to produce 5,000 results to merge. Use a cursor instead of an offset.
- Making the search index the source of truth. It is derived data and it will drift.
- Writing separately from the application to both the database and the index.
- Deep pagination with large offsets, which gets much worse as shard count grows.
- Splitting a small index into too many shards, making every query slower for nothing.
The a search index follow-up an interviewer will ask
"A user creates an item and immediately searches for it. Is it there?"
Probably not, and it is better to say so upfront. Search indexes are near-real-time. The document has to be indexed and the index refreshed before it is visible, usually about a second, and longer if the indexing pipeline is behind. So the design should not rely on search for that moment. If the user is looking at their own newly created item, serve that view from the database, which knows immediately. If they land on a search results page, I can merge their pending item into the results on the client, or force a refresh of just that one document, which is expensive and fine at low volume. What I would avoid is forcing a global refresh after every write, because that destroys indexing throughput. The general principle: search is derived, eventually consistent data, and any path where a user must see their own change immediately should read the source of truth instead.
Your turn
Why can a search index find "running shoes" when the document says "run shoe", but a database LIKE query cannot?
Learn it from the source
This page gets you interview-ready. These take you to where the idea is actually specified or measured — read them when you want the real depth.
- Apache Lucene — Documentation ↗The inverted-index engine underneath Elasticsearch and Solr.
- Elasticsearch — Reference ↗Analysis, mapping and relevance scoring as operational concerns.