Semantic search: the part nobody wants to build
LLM-powered product search is the easiest demo in ecommerce AI and the hardest thing to ship into production. Here is why.
Semantic product search is the easiest demo in ecommerce AI. Four hours with OpenAI embeddings and a hacky vector store and you have a live demo your marketing team will want to show everyone. It is also, genuinely, the hardest thing in ecommerce AI to ship into production.
The gap between demo and production
- The demo uses a thousand products. The production catalogue has a million.
- The demo assumes English. The production storefront serves four languages.
- The demo re-embeds every time. The production PIM ships updates by the hour.
- The demo is a search box. The production version has faceted filters, sort orders, promo pinning, and a merchandising team who will email you.
- The demo answers queries in 800ms. The production version gets told to answer in 80ms.
Every one of those gaps is a month of work. You cannot close them with a bigger prompt.
What I build instead
Hybrid search. Lexical (Typesense, Algolia, Meilisearch) handles the exact-match traffic that dominates ecommerce — people type a SKU, a brand name, a size — and they want an instant answer. Semantic (embeddings, vector search) picks up the rest. A fallback blender sits in front and decides which result set to return based on confidence scores.
“If the user typed a product name, do not call an LLM. If they typed a problem, do.”— The rule on every semantic search project I have shipped
Embedding pipelines get messy at scale. A million-SKU catalogue is not re-embedded on the fly. I batch the deltas through Horizon, store the vectors in pgvector alongside the product data, and re-embed on a schedule rather than on every write. The merchandising team gets a manual re-index button for promotional moments.
The honest verdict
Semantic search moves the needle. On the two catalogues where I have shipped it, zero-result queries dropped by around thirty per cent and "long-tail" conversion lifted by a meaningful margin. But the payoff arrives at the end of a long build, not at the end of a demo. Budget accordingly — or do not start.