Embeddings & RAG

"Walk me through how RAG works and when you'd choose it over fine-tuning."

This is the core interview question for retrieval-augmented systems. Interviewers check for all 5 beats below — missing one draws a follow-up probe, missing two reads as "has only read about RAG, never built one."

Miss one and they probe.

1. What RAG does — grounds responses in your own data by retrieving at query time, instead of relying on training memory
2. The pipeline — chunk → embed → store → retrieve → augment → generate
3. Why it beats fine-tuning for dynamic/private data — no retraining, data stays current
4. Key failure modes — chunk size, retrieval quality, context overflow, stale embeddings
5. When fine-tuning wins instead — changing model behavior/style, not what it knows

"For frequently updated internal API docs, I'd choose RAG over fine-tuning — weekly retraining would be prohibitively expensive. I'd chunk the docs at around 400 tokens with 50-token overlap, embed each chunk, and store them in a vector store. At query time, I'd embed the question, retrieve the top-3 most similar chunks, inject them into the prompt with a strict 'answer only from context' instruction, and generate the response. The main failure mode to watch is stale embeddings — any document update needs immediate re-indexing. I'd revisit fine-tuning only if I needed to change the model's writing style, not just its knowledge."

Your version doesn't have to match word-for-word — all five beats must be present.

• "Walk me through how RAG works" — asked in nearly every AI engineering interview
• "When would you use RAG vs. fine-tuning?" — the decision-rule follow-up
• "What can go wrong with a RAG system?" — system-design interviews
• Naming the six pipeline steps and the failure modes separates senior from junior answers

• Chatbot can't answer questions about internal docs → no retrieval step, model only has training memory
• Retrieval returns irrelevant chunks → chunking too large/mixed-topic, or missing metadata filters at scale
• Answers cite outdated info after a doc update → stale embeddings, no re-index
• Model answers confidently even when retrieval fails → missing "answer only from context" instruction

An embedding is a list of numbers encoding a text's meaning — similar meanings produce numerically close vectors. That's why "reset password" and "account recovery" score as similar even though they share no words. RAG uses this: retrieve the most relevant chunks by meaning at query time, inject them into the prompt, and get the model to answer from your actual content instead of guessing from training memory.

Chunk → Embed → Store → Retrieve → Augment → Generate. Every design decision in a RAG system — chunk size, overlap, top-k, metadata filters — is a choice about one of these six steps. If you can't name all six in order, you can't debug a RAG system that's returning bad answers, because you don't know which step to inspect.

An embedding is a list of numbers (a vector, often 1536–3072 of them) that represents the meaning of a piece of text — its position in a high-dimensional space encodes what it means, not how it's spelled.

Like GPS coordinates for meaning: "reset password" and "account recovery" land in the same neighborhood even with zero shared words.

Measures the angle between two vectors, not their length. Close to 1.0 = same direction = similar meaning. Preferred over Euclidean distance because document length doesn't distort the score.

Like comparing two compass headings instead of two walking distances — a one-paragraph summary and a full article on the same topic point the same direction even though one "walked further."

Chunking splits a document into smaller pieces so each covers one topic and fits a retrieval result — typically 300–500 tokens. Chunk overlap (typically 50 tokens) repeats a buffer of text between adjacent chunks so a sentence straddling a boundary isn't split across two incomplete chunks.

Like cutting a rope into overlapping segments instead of clean cuts — nothing important gets severed at exactly the cut point.

A vector store is a database optimized for querying vectors — from a plain Python list for prototypes to Pinecone/Weaviate at production scale. Top-k retrieval returns the k chunks with the highest similarity to the query — k=3–5 is typical: enough context without overflowing the context window.

Like a librarian handing you the 3 most relevant books instead of the whole library — too few and you miss the answer, too many and you drown in it.

The step where retrieved chunks are inserted into the prompt before the user's question, paired with an instruction to answer only from that context.

Without the "only from context" instruction, the model may read your retrieved chunk and then answer from memory anyway — the instruction is what actually forces it to stay grounded.

A production failure mode: a document is updated but its vector-store entry isn't re-indexed. The old chunk keeps getting retrieved and the model answers from outdated content.

Like a library catalog card that still points to a book's old edition — the shelf changed, the index didn't.

Choose RAG when data changes often, you need an audit trail to source documents, or the knowledge is private and never in training data. Choose fine-tuning when you need to change how the model behaves — style, tone, domain vocabulary — not what it knows.

RAG swaps out the reference book on the shelf; fine-tuning retrains the person reading it. Don't retrain the person every time the book gets a new edition.

Top-k retrieval degrades as a document set grows — at 50,000 documents, the "5 most similar" chunks may only be loosely related. Fix: add metadata filters (department, date range, category) before similarity search, narrowing the space, then rank by cosine similarity within that subset.

Like searching within a filtered folder instead of the entire filing cabinet — you rank the right shelf, not the whole building.

Self-assessment, not graded. Answer before checking your notes.

1. What is an embedding?

• A compression technique that reduces file sizes
• A list of numbers that encodes the meaning of a piece of text
• A way to translate text from one language to another
• A method of encrypting sensitive data

2. Why is cosine similarity preferred over Euclidean distance for embedding-based retrieval?

• Cosine similarity is faster to compute
• Cosine similarity is not affected by document length — only by the direction of meaning
• Euclidean distance only works with images, not text
• Cosine similarity returns values between 0 and 1, which is easier to interpret

Self-assessment, not graded.

3. What is the correct order of steps in a RAG pipeline?

• Generate → Retrieve → Augment → Embed → Chunk → Store
• Chunk → Embed → Store → Retrieve → Augment → Generate
• Embed → Chunk → Retrieve → Store → Generate → Augment
• Store → Chunk → Embed → Retrieve → Generate → Augment

4. What is the purpose of chunk overlap in a fixed-size chunking strategy?

• To reduce the total number of chunks needed
• To ensure that the LLM sees more context during generation
• To prevent key information at chunk boundaries from being split across two incomplete chunks
• To increase the cosine similarity scores between chunks

Self-assessment, not graded.

5. A RAG-powered chatbot consistently returns irrelevant answers for "billing" questions — it keeps retrieving sections about "shipping" instead. What is the most likely root cause?

• The LLM's context window is too small
• The embedding model has a knowledge cutoff and doesn't know about billing
• The billing content is split across chunk boundaries, or the chunks are too large and contain mixed topics
• The system prompt is missing the "Answer ONLY from context" instruction

6. Your team has internal API documentation that changes every week and you want to build a Q&A chatbot over it. Which approach would you choose, and why?

• Fine-tuning — it gives the model more accurate domain knowledge
• RAG — because the data changes frequently and RAG doesn't require retraining
• Prompt stuffing — inject the full documentation into every request
• Neither — LLMs cannot answer questions about private internal data

Self-assessment, not graded.

7. Which of the following would NOT improve retrieval quality in a RAG system with poor results?

• Reducing chunk size from 1000 tokens to 400 tokens
• Increasing the chunk overlap from 0 to 50 tokens
• Switching the LLM from one model to another
• Adding metadata filters (e.g. department, date range) before similarity search

8. Interview-style: which answer for "Walk me through how RAG works" is strongest — one naming the pipeline + failure modes + the fine-tuning decision rule, or one just saying RAG "adds more context" and is "always better" than fine-tuning? Explain why in 2 sentences.

≈45 min · Python + an LLM API key

Goal: given a query, find the most semantically similar document from a small library — the core mechanic every RAG system depends on.

Using your Session 02 prompt-library.md as the document set, build a list of document dicts (title, when-to-use, prompt, parameters) and a find_best_template(query) function. Test it with 5 queries covering different templates (e.g. "I need to classify support tickets automatically", "My output format keeps changing between runs").

Write down: for each of the 5 queries, which template matched and whether it was actually the right one — at least 4 of 5 should be correct.

≈45 min · cements the "answer only from context" instinct

Goal: turn a retrieved chunk into a specific, grounded answer instead of a generic one — and see what happens when nothing matches.

Using the retrieval function from Assignment 1, write a function that takes the best-matching template and asks the model to explain how to apply it to the user's specific query, with a system instruction to answer only using the provided template. Run it on 3 queries, including one that shouldn't match any template well (e.g. "write me a poem").

Write down: whether each explanation was specific to the query (not generic), and whether the no-match case honestly acknowledged the limitation instead of hallucinating an answer.