Part 1 of 6
Foundations of a Production Restaurant Recommender
What we'll build: A personalized restaurant-suggestion engine, from scratch: a city's restaurant catalog, an LLM pipeline that turns real customer reviews into structured attributes and a written portrait per place, semantic search over those profiles, and a per-user taste profile driving a personalized feed. And because none of it can be trusted unmeasured — an eval suite that grades the whole thing.
TLDR: Production AI is not "send the question to a model" — it is retrieve, rank, generate: a database narrows thousands of candidates cheaply, and the model only reasons over what it was handed. This part teaches the retrieval half from zero: embeddings (lists of numbers encoding what a text means), why a query and a document must be embedded differently or search quietly returns generic matches, and why "gluten-free" must never be answered by similarity — vibes get vector search, dealbreakers get SQL WHERE clauses. It closes with the fusion trick that merges keyword and vector rankings using only positions, and the capacity math showing a whole city's vectors fit in about twenty megabytes of the PostgreSQL you already run — no dedicated vector database needed.
Ask an LLM "How is Beaver, near Konyaaltı beach in Antalya?" and you'll get something no star rating can give you: "cozy place, mostly young crowd, good for casual evenings." Not a 4.3 and a wall of unread opinions — an actual answer.
This series documents building that capability properly: a production restaurant recommendation system on real AWS infrastructure, with real cost constraints, headed for real users. The stack is a Java/Spring GraphQL API next to a Python ML service, PostgreSQL with the pgvector extension for retrieval, and Claude for the language work. One city to start: Antalya, a few thousand restaurants.
Part 1 covers the foundations — the concepts that decide the architecture before any architecture exists.
🎯 Why "just ask the LLM" doesn't survive production
The naive version — send the user's request straight to an LLM — fails on four counts:
Hallucination. Ask for restaurant suggestions and some of the names in the answer will not exist. This isn't a bug awaiting a patch; it's the nature of a model generating plausible text without checking it against anything.
Freshness. The model has no idea what opened or closed last month.
Memory. It knows nothing about what this specific user loves.
Cost control. Every answer costs whatever the provider charges, on every request, forever.
The industry's answer is RAG — Retrieval-Augmented Generation — and it's less exotic than it sounds: keep the facts in your own database, retrieve the relevant records first, and let the LLM reason only over what it was handed. The LLM stops being the source of truth and becomes a language layer over your data. It can't invent a restaurant it was never shown.
Nearly every production AI system reduces to the same three-stage pipeline:
RETRIEVE — narrow thousands of candidates to about 30, cheaply, with a database RANK — spend real compute ordering those 30 for this specific user GENERATE — write the human-facing explanation ("cozy meyhane with a sea view — matches your love of quiet local spots")
Hold onto that pipeline; every section below slots into it.
One more framing decision. Netflix and Spotify run on collaborative filtering — "users like you also liked this" — learned from millions of user-item interactions. A new product has no interactions to learn from (the cold-start problem), so the approach flips: describe the items richly and match them against what users say they love. That's content-based recommendation, and it's why this system invests so heavily in deep restaurant profiles rather than a user-user interaction matrix.
🧮 Embeddings, explained the way I wish someone had explained them to me
An embedding is a list of numbers representing what a text means. An embedding model is a neural network trained on billions of sentence pairs until similar meanings produce similar number lists. The list is called a vector; the model used here outputs 1024 numbers per text.
Easier to see in two dimensions:
"romantik deniz manzaralı restoran" → [0.92, 0.15]
"quiet seaside dinner spot" → [0.89, 0.21]
"cheap fast-food burger joint" → [0.11, 0.97]
The first two point in almost the same direction — even though one is Turkish and one is English. Meaning survives translation; direction encodes meaning. Similarity is measured with cosine similarity: the cosine of the angle between two vectors. 1.0 = same meaning, 0 = unrelated. Compute one by hand once (it's multiplication and addition) and the mystery evaporates permanently.
For a Turkish travel product this multilingual property is the whole game: a Turkish query must find restaurant profiles that may be written in English. It's also why model choice matters — this system needs an embedding model trained multilingually, not one optimized for English only.
❓ "Does it embed token by token, or one vector for the whole sentence?"
Both, at different layers. Internally the model splits text into tokens (word-pieces) and produces a vector per token — then a final step called pooling (usually just averaging) collapses them into one. Three words in or three paragraphs in: exactly one 1024-number vector out.
That creates a real design constraint: compress 10,000 words into one vector and every detail dilutes into mush. It's why the RAG world practices chunking (split long documents, embed each piece), and why the text embedded per restaurant here is capped at roughly 250 words — the comfortable payload for a single vector.
❓ "If both sides become vectors, isn't a query just a small document?"
That was my assumption too, and it fails in a subtle way. Watch what an embedding model does with these three texts if you embed them all the same way:
A) "romantic place" ← the user's query
B) "a nice spot for couples" ← some short, generic sentence
C) [250-word portrait of a quiet, candle- ← the restaurant that actually
lit restaurant above the sea] answers the query
The model puts A closest to B. Why? Because embeddings measure similarity — and A and B genuinely are similar: both short, both vague, both wish-like. C is a completely different kind of text: long, concrete, descriptive. But C is the one the user wants. What retrieval needs isn't "which text resembles my query" — it's "which text answers my query." Similarity and relevance are different questions, and a plain embedding only knows the first one.
The failure is invisible in production: nothing throws, nothing logs. Result #1 just becomes a generic match instead of the right restaurant, and the right answers quietly slip out of the top of the list — no alarm attached.
The fix: models like Cohere's are trained on millions of real (query → the document that answered it) pairs, so they learn both roles — but you have to tell them which role your text is playing. That's the input_type flag: search_document when storing content, search_query when searching. Think of it as translation: search_query mode takes "romantic place" and effectively asks "what would the description of this wish look like?" — then places the vector there, in description-space, right next to portrait C. Same text, different flag, genuinely different vector out.
Two engineering rules fall straight out of "different flag = different vector":
- Never mix. Every vector stored in the index must come from the same model and the same flag — otherwise you're comparing coordinates from two different maps.
- Cache by flag. The embedding cache key must include the flag (hash(model + input_type + text)), or a vector from the wrong mode gets served and you're back to the silent failure.
In this codebase input_type is a required, validated parameter on every call — because a flag someone can forget is a production incident on a delay timer.
❓ "OK, but what exactly do you embed? The restaurant's JSON?"
This question decides more retrieval quality than the index or the model. Each restaurant in this system has rich structured data:
{ "vibe": {"cozy": true, "lively": false},
"crowd": {"age_profile": "young"},
"ambiance": {"noise_level": "quiet", "view": "sea"},
"dietary": {"gluten_free_options": false} }
The tempting move is to embed that JSON directly. Don't. Embedding models are trained on natural language — sentences, paragraphs, human text. Feed them JSON and the braces, quotes and keys eat tokens while carrying no meaning; worse, values like "lively": false or "gluten_free_options": false barely register in meaning-space at all (what direction does false point?). The vector comes out mushy, and — the familiar theme — nothing errors. Quality just drops.
The pattern that works is one fact, two representations:
- JSON stays JSON — it powers SQL hard filters, the UI, and analytics. Structure is what databases are good at.
- For the vector, write prose. Each restaurant gets a roughly 250-word natural-language portrait — "A quiet, candle-lit spot above the sea in Konyaaltı. Draws a mostly young crowd for slow dinners; conversation-level noise even on weekends..." — written by an LLM from the structured attributes and their supporting evidence. That text is what gets embedded, because it's the same kind of text the embedding model grew up on, and the same kind of language users type when they search.
A useful way to think about it: the portrait is written for the embedding model as its reader. Dense with searchable meaning (vibe, crowd, occasion, signature dishes), zero marketing fluff ("an unforgettable culinary journey" earns no vector direction worth having), and capped at the length one vector can carry before diluting.
❓ "Restaurants change. Models change. Do you just... re-embed everything every time?"
A vector is a derived value — derived from two things: the text it came from, and the model that computed it. The moment either changes, the stored vector is silently stale. And "silently" is the theme of this whole article: a stale vector doesn't error, it just ranks wrong.
So every vector in this system carries two small companions:
A fingerprint of the source. When a restaurant's portrait is generated, we hash the material it was built from and store that hash next to the vector. On the next pipeline run, we hash the current material and compare. Same fingerprint → nothing changed → skip, pay nothing. Different → regenerate the portrait, re-embed, overwrite. This one cheap trick turns "re-process the whole catalog nightly" into "re-process the handful that actually changed" — and it makes the pipeline safe to re-run at any time, which matters more than it sounds: batch jobs crash, and a crashed job that can simply be restarted is worth ten pages of recovery logic.
A model label. Each vector records which embedding model produced it. Remember the rule from the input_type section — vectors from different models are coordinates on different maps, never comparable. That rule has a lifecycle consequence: the day we upgrade to a newer embedding model, we can't upgrade "gradually" — a half-migrated index would be comparing two maps at once. The label makes the migration honest: backfill every vector with the new model (tagged as such), verify quality on the new set, then flip retrieval over in one move. The label also answers the auditor's question six months later: which model produced the vector behind this result?
None of this is AI magic — it's cache invalidation, the oldest hard problem in the book, wearing a new costume. A vector is a cache of meaning, and caches need keys.
🗺️ Finding the nearest vectors fast — HNSW
Retrieval now means: given a query vector, find the K stored vectors with the smallest cosine distance. Nearest-neighbor search.
An honest number first: at our scale — around five thousand restaurants, which is not much — brute force, checking every vector, takes single-digit milliseconds. At this scale no index is needed. At a million vectors it would be, and the standard tool is HNSW (Hierarchical Navigable Small World).
The mental model is a highway system 🛣️ — the top layer of the graph has few nodes with long-range links (intercity highways); lower layers are denser and local (city streets). A search enters at the top, hops greedily toward the query, drops a level, refines — like zooming into a map. A few hundred comparisons instead of millions.
The tradeoff is in the word approximate: it can miss neighbors. The metric is recall@K — of the true top-K nearest, how many did the index actually return? The index gets built here anyway (it's free at this size), and recall gets measured against exact scan — because a measured number beats a quoted blog post in any technical discussion.
🧱 Where vectors fail: loud Friday, quiet Tuesday, celiac always
The best stress test of this design came from a friend, in one Slack message:
"Friday night it's a loud, long meyhane dinner with old friends. Tuesday it's a quiet corner, a solo lunch and a book. And my partner has celiac — anywhere we go together, gluten-free isn't a preference, it's medical. Same me, three completely different tables. Will your thing understand all of them?"
Three separate lessons in one message.
1. The Friday and Tuesday parts are embeddings at their best. Indirect, mood-heavy text ("loud, long meyhane dinner") maps cleanly into lively-open-late territory in meaning space; "a quiet corner and a book" lands in calm-daytime territory. One person, two moods — and no single similarity score could serve both. None has to: each query carries its own context and finds its own region. Semantic search, working as designed.
2. Celiac must never be handled by similarity. Vector search returns closest matches — and an ordinary bakery-café can be 90% similar to a dedicated gluten-free one along every other dimension. For a celiac guest, "close" is a catastrophe: a wrong answer here isn't a bad recommendation, it's harm. Constraints like gluten-free, vegan, open-now, within-2-km are hard filters: SQL WHERE clauses that guarantee, not suggest.
Probabilistic tools for vibes. Deterministic tools for dealbreakers. Knowing which is which is half the job.
3. Exact words still matter. A search for "künefe" had better return the place whose written portrait literally says künefe. Embeddings blur specifics; classic full-text search with Turkish stemming ("manzara / manzaralı / manzarası" all match) nails them.
So production retrieval is hybrid: run vector search AND keyword search side by side (hard filters on top of both), and you end up with two separately ranked lists for the same query:
Vector search says: Keyword search says:
1. Beaver (cosine 0.83) 1. Künefeci Ali (ts_rank 12.4)
2. Vanilla (cosine 0.79) 2. Beaver (ts_rank 8.1)
3. Künefeci Ali (cosine 0.71) 3. Seraser (ts_rank 3.2)
Now you have to merge them into one list — and here's the trap: the scores can't be compared. Cosine lives between 0 and 1; keyword rank scores are on some arbitrary scale. Is cosine 0.83 better than ts_rank 12.4? (ts_rank is Postgres's full-text relevance score; its unit is arbitrary — which is exactly the problem.) The question has no answer — they're measurements from two different instruments.
Reciprocal Rank Fusion (RRF) solves this by throwing the scores away and using only each result's position. Every restaurant earns points from each list: 1 / (60 + its position). Add them up, sort by total:
Beaver : 1/(60+1) + 1/(60+2) = 0.0164 + 0.0161 = 0.0325 ← winner
Künefeci Ali : 1/(60+3) + 1/(60+1) = 0.0159 + 0.0164 = 0.0323
Vanilla : 1/(60+2) + 0 = 0.0161 (only in one list)
Beaver wins because it ranked high in both lists — which is exactly the behavior you want: agreement between two independent signals beats excellence in one. The constant 60 is just a dampener from the original paper (it keeps a #1-vs-#3 gap from being overly dramatic); nobody tunes it. Positions are comparable across any two lists, scores never are — that's the whole trick, and it's three lines of SQL.
💾 The vector database that wasn't needed
Dedicated vector databases are their own product category now: Pinecone, Qdrant, Weaviate, Milvus — plus managed options like AWS OpenSearch. What are they? Databases purpose-built to store vectors and answer "find the nearest N" extremely fast, at extreme scale.
The key word is scale. They earn their complexity when you have tens of millions of vectors, thousands of queries per second, or strict tenant isolation. Before adopting one, do the capacity math for your actual workload:
around 5,000 restaurants × 1024 dimensions × 4 bytes ≈ 20 MB
Twenty megabytes. Smaller than the images attached to this post. Meanwhile AWS's managed option (OpenSearch Serverless) bills in compute units with a monthly floor in the hundreds of dollars — for capacity this workload would use a fraction of a percent of. (AWS did ship a scale-to-zero tier in 2026, but a user-facing search service is never idle — sustained traffic keeps the compute units warm, and the bill lands right back where it started.)
The alternative: pgvector, an extension for PostgreSQL — the database already running, already backed up, already paid for. One CREATE EXTENSION vector; and Postgres understands vectors, cosine distance, and HNSW indexes.
And the underrated win isn't cost. It's that one SQL query now does everything — hard filters, geo radius, Turkish full-text, vector similarity — in one place, in one transaction. No second datastore, no sync pipeline, no drift between "the real data" and "the search data."
The lesson generalizes: do the capacity arithmetic before choosing infrastructure. Ten minutes of multiplication saved a service, a pipeline, and a four-figure annual bill.
⏭️ Next
Foundations set: the retrieval layer's shape and the reasoning behind it. Part 2 covers the architecture — why the ML code lives in a Python service beside a Java platform, how the two share one database without stepping on each other — and the part nobody hands you: names, locations and opening hours are the easy half of a restaurant catalog. The valuable half — "vibe": "cozy, mostly young crowd" — doesn't exist anywhere as data. We build that attribute layer in-house, with an LLM pipeline, evidence requirements for every claim, and a hard budget cap so the pipeline can't surprise anyone's credit card. That's the next post.
For backend engineers eyeing this field: it's databases, indexes, trade-offs and cost math — the work you already do — plus one genuinely new primitive worth learning properly.
The vector is the new column.