RAG Pipeline

Document Chat proved an LLM can answer questions about your data — as long as everything fits in one request’s context window. This project removes that ceiling: instead of resending a whole document on every question, you build a real retrieval pipeline that finds and injects only the relevant pieces of a much larger document set, on demand.

What you’re building

rag-pipeline ingests a folder of documents, chunks and embeds them into a persistent vector index, and at query time retrieves only the chunks above a similarity-score threshold — not the whole corpus. It ships with an A/B mode that runs the same question through the model with and without retrieved context, so you can see exactly what retrieval is buying you on a question-by-question basis, instead of assuming it’s working because the answers sound reasonable.

The RAG pipeline

Ingestion (top) and querying (bottom) are two separate paths that share one index. A stale index — documents changed without re-running the top path — is one of the most common ways this pipeline quietly starts giving wrong answers.

Core concepts, three levels deep

1. Embeddings

Definition: a list of numbers (a vector) that encodes the meaning of a piece of text, not its exact wording. Texts with similar meaning produce numerically close vectors, which is what makes searching by meaning — instead of by keyword — possible at all.
In this project: every chunk you ingest, and every query a user sends, gets converted to a vector before anything else happens. Retrieval is entirely a numeric comparison over these vectors — the model never re-reads your raw documents at query time, it reads whatever chunks the vector comparison surfaced.
Practical consequence: two phrases that share zero words (“reset password” and “account recovery”) can still retrieve correctly, because embeddings capture intent, not literal text overlap. That’s the entire reason this beats a keyword search.

2. Chunking and chunk overlap

Definition: splitting a document into smaller pieces so each one covers roughly one topic and fits comfortably in a retrieval result — typically 300-500 tokens, with a 50-token overlap buffer at each boundary.
In this project: the chunk size and overlap you choose directly determine what’s retrievable. A fact that straddles two chunks with no overlap can end up incomplete in both of them — retrievable in neither.
Practical consequence: when retrieval looks wrong, check chunking before you touch anything else. Bad chunk boundaries are one of the most common — and most fixable — causes of a RAG system returning incomplete or off-topic results.

3. Retrieval: cosine similarity, top-k, and score thresholds

Definition: cosine similarity ranks chunks by the angle between their vector and the query’s vector (direction, not magnitude); top-k returns the k highest-scoring chunks; a score threshold discards anything below a “similar enough” cutoff even if it’s in the top-k.
In this project: these three settings together decide what actually reaches the model. Top-k alone can return weak matches if nothing in the index is truly relevant — the score threshold is what lets the pipeline say “nothing in my documents answers this” instead of forcing a bad match into the prompt.
Practical consequence: cosine similarity — not Euclidean distance — is the right metric here specifically because it isn’t skewed by chunk length, so a short chunk and a long chunk about the same topic can still score as equally relevant.

4. Grounding and index freshness

Definition: grounding is instructing the model to answer only from retrieved context (and say so when it can’t); index freshness is whether the vector store reflects the current state of your source documents.
In this project: without an explicit “answer only from the provided context” instruction, the model can quietly fall back on training memory instead of your actual documents — which defeats the entire point of building this pipeline. Separately, if a source document changes and you don’t re-embed it, the old chunk stays in the index and keeps getting served as if it were current.
Practical consequence: treat re-indexing after a document update as part of running this pipeline, not an optional maintenance step — a stale embedding is a wrong answer delivered with full confidence, which is exactly the failure mode citations and grounding are supposed to prevent.

Decision rules

If…	Then…
A document set comfortably fits in one context window and rarely changes	Stuff it into context directly — that’s Project 1’s approach; you don’t need retrieval infrastructure for this
The document set is large, spans many files, or needs to scale past one context window	Build retrieval (this project) — chunk, embed, index, and retrieve only what’s relevant per query
Source data changes frequently and answers need a traceable source	Choose RAG over fine-tuning — no retraining needed, and the index can be updated independently of the model
You need to change the model’s writing style, tone, or domain vocabulary, not its knowledge	Choose fine-tuning over RAG — that’s a behavior change, not a knowledge-injection problem
Retrieval quality degrades as the corpus scales up	Add metadata filtering before similarity search — don’t just raise top-k, which gets noisier, not better, at scale

Common mistakes

Zero or too-small chunk overlap, silently splitting the sentence that contains the answer across two incomplete chunks.
Skipping the “answer only from context” instruction, so the model quietly falls back to training memory instead of the chunks you actually retrieved — defeating the point of retrieval.
Never re-indexing after a document update, so the pipeline keeps confidently citing outdated content indefinitely.
Reaching for a bigger top-k instead of metadata filters when retrieval quality drops at scale — a larger candidate pool without narrowing it first just adds more noise.

Key concepts at a glance

Concept	One-line definition	Why it matters for `rag-pipeline`
Embedding	A vector that encodes the meaning of a piece of text	The unit everything in this pipeline — chunks and queries alike — is compared as
Chunking + overlap	Splitting documents into retrievable pieces with a boundary buffer	Determines whether a fact is actually retrievable at all
Cosine similarity / top-k / score threshold	Angle-based ranking, capped at k results, filtered by a relevance cutoff	The exact mechanism this project’s retrieval step runs on
Index freshness	Whether the vector store reflects the current source documents	Why re-indexing on update is part of operating this pipeline, not optional maintenance

Go deeper: Embeddings & RAG — the full concept walkthrough (vectors, cosine similarity, the six-step pipeline, RAG vs fine-tuning) →

RAG is the single most-asked-about AI Engineer skill in job descriptions today — this is the project that lets you speak to it with specifics, not buzzwords.

Clone this project, run it locally, and compare your output against the description above.

RAG Pipeline on GitHub ↗

Self-check only — no score is recorded anywhere. Answer each question yourself, then click to reveal.

1. Your rag-pipeline's A/B mode shows the RAG answer quoting a paragraph almost verbatim from an ingested document, while the no-RAG answer to the same question is fluent but generic. What does that comparison actually prove?
- a) The RAG answer is grounded in a specific chunk retrieved from your index; the no-RAG answer came entirely from the model's training memory, with nothing to verify it against
- b) RAG always produces better-written prose than no-RAG
- c) The embedding model is malfunctioning whenever the two answers differ
- d) A/B mode proves the vector store used exactly 5 chunks
Reveal answer

A. Quality of prose isn't the point — traceability is. The RAG answer can be checked against a real chunk in your index; the no-RAG answer is a claim you have to trust, with no source to point to. That gap is exactly what A/B mode exists to make visible.
2. You chunk a 40-page internal doc into 300-token pieces with zero overlap. A user's question has an answer whose key sentence straddles the boundary between chunk 12 and chunk 13. What happens, and why?
- a) Retrieval works fine — embeddings represent the whole document, not individual chunks
- b) Retrieval likely returns an incomplete answer, because with zero overlap the boundary sentence is split across two chunks and neither one alone contains it
- c) Increasing top-k to 50 fixes this regardless of chunking
- d) Cosine similarity automatically rejoins split sentences at query time
Reveal answer

B. Chunk overlap exists precisely for this case — a fixed buffer of repeated tokens at each boundary so a sentence that straddles two chunks is still fully present in at least one of them. Zero overlap means boundary information can go missing from every chunk simultaneously, which no amount of top-k tuning fixes after the fact.
3. In 1-2 sentences: why does this pipeline rank chunks by cosine similarity instead of raw (Euclidean) distance between vectors?

Reveal answer

Model answer: cosine similarity measures the angle between two vectors — direction, not magnitude — so a short chunk and a long chunk about the same topic still score as similar. Euclidean distance is sensitive to vector length, which would unfairly penalize longer chunks purely for being longer, regardless of whether their meaning matches the query.
4. Your index has 5,000 chunks in dev and retrieval looks great. In production, after ingesting 500,000 chunks across many unrelated document sets, top-k results for "billing" questions start mixing in unrelated "shipping" content. Your score-threshold filter alone doesn't fix this. What actually helps?
- a) Raising the generation temperature
- b) Switching to a bigger embedding model — that alone always fixes retrieval at scale
- c) Adding metadata filtering (department, date range, document set) before similarity search, so the corpus is narrowed before top-k ever runs
- d) Increasing max_tokens on the generation call
Reveal answer

C. Top-k similarity search gets noisier as the candidate pool grows — at half a million chunks, the "5 most similar" can be only loosely related even with a score threshold. Narrowing the search space with metadata filters before ranking is what keeps retrieval precise at scale; temperature, model size, and max_tokens don't touch the retrieval step at all.
5. A source document in your index gets updated with new pricing, but your pipeline keeps citing the old price a week later. Name the failure mode and the fix.

Reveal answer

Model answer: this is a stale embedding — the vector store was never re-indexed after the source document changed, so the old chunk is still what gets retrieved and cited. The fix is to re-index the changed document immediately on update (and, for a stricter guarantee, add a timestamp filter so only recently-indexed chunks are eligible for retrieval).
6. When would you reach for fine-tuning instead of this project's retrieval approach?
- a) When your source data changes weekly and you need an audit trail back to the original document
- b) When you need to change how the model behaves — its writing style, tone, or domain vocabulary — rather than inject new facts
- c) When you want zero training cost and instant updates
- d) When your corpus is too large to fit in a vector store
Reveal answer

B. RAG and fine-tuning solve different problems. RAG keeps the model's *knowledge* current without retraining — the right call for data that changes and needs a traceable source. Fine-tuning changes the model's *behavior*, and is expensive to redo every time the underlying facts change, which makes it the wrong tool for frequently-updated knowledge.

These three tasks force you to verify retrieval is actually working correctly — chunk boundaries, grounding, and staleness — instead of assuming a plausible-looking answer means the pipeline is right. This is separate from the "What to do" tab's clone-and-run pointer: that gets the reference implementation running; this is what to build and verify yourself.

1. Chunk-boundary check — before you trust retrieval
30-45 min

Take a real multi-page document (a spec, a policy doc, notes from your own work). Find one fact whose key sentence would land right at a chunk boundary if you split the document into fixed-size pieces. Chunk it at 300-500 tokens with 50-token overlap, then deliberately re-chunk a copy with zero overlap. Query both versions for that fact and write down what happens to each — this is the cheapest way to see chunk overlap matter before it costs you a real wrong answer.
2. Ship the retrieval + grounded generation loop
90-120 min

Build the pipeline: chunk a folder of real documents, embed each chunk, store them in a persistent index, embed incoming queries, retrieve the top-k chunks above a score threshold, and generate an answer using a system prompt that instructs the model to answer only from the retrieved context — and to say so explicitly when nothing above the threshold is relevant, instead of guessing. Wire up the A/B mode so the same query also runs with no retrieved context at all, and print both answers side by side.
3. Run the A/B comparison, then break freshness on purpose
45-60 min

Run 3 real questions through A/B mode and note where the no-RAG answer sounds confident but has no traceable source, versus where the RAG answer's citation actually checks out against your indexed documents. Then update one source document's content and query it again without re-indexing — confirm your pipeline keeps serving the stale answer. Write two sentences on what would need to change architecturally to detect and fix that automatically, rather than relying on someone remembering to re-index by hand.

Deliverable

A short write-up with: Task 1's chunk-boundary result (overlap vs. no-overlap), Task 2's working pipeline (or a link to it), and Task 3's three A/B comparisons plus the staleness observation and your two-sentence fix note.