How to Build AI-Powered Natural Language Search for Ecommerce

You have a large catalog of hundreds of thousands of products and want your customers to easily search them. If someone searches for "black leather sneakers", standard database search works fine; it matches the exact words. But if they search for "something light to wear to an outdoor summer wedding in Italy", standard search breaks down. No product title says "Italy," and very few dresses are tagged with "summer wedding."

To solve this, you need to build an AI Search. Instead of matching literal strings, it uses Large Language Models (LLMs) to translate human intent into structured database filters and rich search descriptions.

Here is how to design, architect, and deploy an end-to-end AI search pipeline for any ecommerce catalog. This guide covers our experience in building a production system for a major ecommerce site with millions of products. Even though the examples are tailored for ecommerce, the architecture works for any AI search over large catalogs of data like articles, videos, songs etc.

1. System Architecture: The Two-Pipeline Strategy

We build two software pipelines that share a single document schema and an embedding model:

• Offline Pipeline (Batch): Enriches, normalizes, and embeds your product catalog ahead of time.
• Online Pipeline (Real-Time): Translates, structures, and embeds the user's search query in under a second.

Why we do it this way

Cost & Latency Optimization: Your catalog does not change every millisecond. You can process 100,000 products offline using cheaper Batch APIs. On the other hand, the shopper is waiting on the site right now, so the search query needs low-latency, real-time endpoints.

The Power of a Shared Schema: The secret to making AI search reliable is forcing both pipelines to speak the exact same language. If the catalog pipeline labels a color as navy but the search query expands it to dark blue, your hard database filters will fail.

2. Offline Step 1: Normalizing Your Product Catalog

We pass every product in your store, including its photos, titles, and raw descriptions, through a Vision-Language Model to extract a standardized English description and strict categorical metadata.

Why we do it

Merchant catalog data is notoriously messy. One brand provides three paragraphs of prose, while another provides only a SKU number and a cryptic color code. By normalizing everything into a strict schema upfront, we eliminate inconsistencies across vendors.

How to implement it

Pass all product context (text and images) into one single prompt per item. Splitting text and image processing into multiple LLM calls increases cost and risks contradictory outputs.

1. Input Format (JSONL for Batching)

Structure your input payloads carefully. Notice that the system prompt instructions come first:

{"custom_id":"sku-10482","method":"POST","url":"/v1/chat/completions","body":{"messages":[
  {"role":"system","content":"<the extraction instructions, byte-identical on every row>"},
  {"role":"user","content":[
    {"type":"text","text":"Title: Adidas Samba OG\nBreadcrumbs: Shoes > Sneakers > Lifestyle\nMerchant copy: The Samba is a football-inspired classic."},
    {"type":"image_url","image_url":{"url":"https://cdn.example.com/10482-front.jpg"}}
  ]}
],"chat_template_kwargs":{"enable_thinking":false}}}

Pro-Tip on Prompt Caching: Keep your system instructions byte-identical across every row in the batch. APIs with prompt caching will bill cached prefixes at a fraction of the standard input rate (for example, ~$0.045 versus $0.18 per 1M tokens), drastically reducing batch indexing costs.

2. Strict Output Structuring (JSON Schema & Enums)

Force the model to return controlled JSON using strict schema constraints. Use enums for all categorical tags so the model does not invent uncontrolled synonyms like "off-white," "cream," or "ecru."

curl -X POST "$BASE/jobs" -H "Authorization: Bearer $TOKEN" \
  -H 'Content-Type: application/json' -d '{
  "input_id": "<input-id>",
  "model": "qwen3.8-27b",
  "reasoning": false,
  "response_format": {"type":"json_schema","json_schema":{
    "name":"product_facets",
    "schema":{"type":"object","properties":{
      "description":            {"type":"string"},
      "google_product_category":{"type":"string"},
      "color":   {"type":"array","items":{"type":"string","enum":["black","white","beige","navy","red"]}},
      "material":{"type":"array","items":{"type":"string","enum":["leather","suede","canvas","mesh"]}},
      "occasion":{"type":"array","items":{"type":"string","enum":["casual","sport","formal","outdoor"]}}
    },"required":["description","google_product_category","color","material","occasion"]}}}
}'

Note on Reasoning: Set "reasoning": false (or disable thinking mode) during batch classification. Extraction tasks gain little benefit from multi-step reasoning, and extra thinking tokens unnecessarily increase output costs.

3. Offline Step 2: Creating Standardized Documents & Embeddings

We format the extracted data into a single text document with a strict block order, and then convert that string into a mathematical vector (an embedding).

Why we do it

Vector models convert entire text blocks into points in high-dimensional space. If you shuffle the order of your fields from item to item, the generated vector changes even if the factual information is identical. Pick one deterministic template and stick to it universally.

How to implement it

1. Standard Document Template

Format the output from Step 1 like this:

Adidas - Samba OG

A low-profile leather trainer with a suede toe cap, gum rubber outsole and
three stripes along the side. A 1950s indoor football shoe now worn as an
everyday lifestyle sneaker.

category: Apparel & Accessories > Shoes
color: white, black
material: leather, suede
occasion: casual
gender: unisex
age group: adult
price: 100 EUR

2. Generating the Index

Send each document to an embedding model:

{"custom_id":"sku-10482","method":"POST","url":"/v1/embeddings","body":{"model":"qwen3-embedding-8b","input":"Adidas - Samba OG\n\nA low-profile leather trainer with...","dimensions":1024}}

Store the resulting vector in a database like pgvector, alongside the extracted tags as traditional, indexed relational columns (for instance, WHERE color && ARRAY['white']).

4. Online Step 1: Query Expansion and Translation

When a user enters a complex or foreign-language search phrase like "something for a summer wedding in Italy", we pass it through a low-latency (realtime) LLM API call before running database queries.

Why we do it

Raw search queries suffer from three key issues:

• They may not be in English (while your catalog embeddings are).
• They express high-level user needs rather than product attributes.
• They are short phrases, whereas vector similarity performs best when matching document-to-document shapes.

How to implement it

The LLM translates and expands the query into the exact same document layout and enum values used by your catalog:

category: Apparel & Accessories > Clothing > Dresses
color: pastel, floral
material: linen, silk
occasion: wedding, formal
gender: female

A lightweight warm-weather dress for an outdoor summer wedding in a hot
Mediterranean climate. Breathable natural fabrics, semi-formal.

Now, the LLM has successfully inferred that an "outdoor summer wedding in Italy" requires a breathable dress with formal/wedding occasion tags, matching your catalog taxonomy perfectly.

5. Online Step 2: Hybrid Filtering and Vector Search

We run a two-phase query against the database:

1. Hard Filtering: Use SQL or database filters based on the extracted metadata attributes, such as category, color, or price.
2. Vector Search: Calculate cosine similarity between the embedded query and the remaining candidate vectors.

Why we do it

Vector search alone is bad at exact math constraints, such as price < $50 or in_stock = true, while relational databases excel at them. Combining hard metadata filtering with vector ranking gives you speed, accuracy, and absolute control over business logic.

How to implement query embedding

Generate the query embedding using the exact same model and dimension configuration used during catalog ingestion:

curl https://api.anex.sh/realtime/v1/embeddings \
  -H "Authorization: Bearer $API_KEY" -H "Content-Type: application/json" \
  -d '{"model":"qwen3-embedding-8b","input":"category: Apparel ...","dimensions":1024}'

6. Online Step 3: Reranking Top Results

Take the top ~50 results returned from the hybrid vector search and pass them through a cross-encoder reranking model.

Why we do it

Vector embeddings excel at retrieving the right candidate set (top 50-100 items), but they often struggle to fine-tune the exact order of the top 10 items. Rerankers are specialized models that score the query directly against each document candidate, ensuring the absolute best matches rise to the top.

Rules for Reranking

1. Rerank against the original raw query: The expanded document is great for broad retrieval, but the user's original query contains the exact nuances and emphasis they typed.
2. Cap candidate lists at 50 items: Reranking adds 300ms to 1000ms of latency. Passing 500 items into a reranker will slow down your search response dramatically without improving performance.

Common Engineering Pitfalls to Avoid

Overly Strict Filtering (The AND Trap)

If your query expander generates 5 different tags, performing a hard AND across all 5 tags often returns 0 results. Fix: Rank tag types by priority (Category > Gender > Occasion > Material). If a query returns fewer than 20 items, progressively drop lower-priority filters.

Model and Dimension Mismatch

If you embed your catalog using 1024 dimensions or Model Version A, but embed incoming queries using 512 dimensions or Model Version B, vector search will silently fail and return garbage scores without throwing an explicit error. Fix: Store the model_id and dimensions parameter as index metadata, and validate them at query runtime.

Free-Text Tag Drift

If you do not enforce enums via Structured Outputs, your LLM will start outputting tags like "crimson", "ruby", and "dark red". A hard database filter on "red" will miss every single one of them.

Format Changes

Changing the document format means re-embedding the whole catalog. That is cheap (see the costs below), but a catalog embedded in one format and queries expanded into another drift apart silently. Fix: Version the document format, and rebuild the catalog and the query expansion from the same version.

Cost Breakdown (Example: 100,000 Products)

By running the catalog normalization pipeline asynchronously via Batch APIs and taking advantage of prompt prefix caching, indexing a massive store remains extremely affordable. The rates are anex.sh batch list prices on 19 October 2026; current rates are on the pricing page.

Because prompt caching cuts instruction costs by up to ~75%, processing an entire 100,000-item catalog costs under $47. Updating individual embeddings after model changes costs pennies ($0.36 per 100k items). The primary operational cost moving forward will simply be the real-time query expansion calls made per shopper search.

Which Models We Recommend

For the offline pipeline, we recommend Qwen 3.8 27B (qwen3.8-27b) for turning product photos and copy into descriptions, and Qwen3 Embedding 8B (qwen3-embedding-8b) for the embeddings. Both run as batch jobs here on anex.sh.

For the online pipeline, we also serve qwen3-embedding-8b as a realtime API, so each query is embedded by the same model as the catalog. To expand the user query, use Qwen 3.8 27B on OpenRouter or any other cheap model you have easily available. The important part is that it needs to have fast response time so make sure to check the time to first token metric and choose the provider and model with the lowest one.