How LLMs Actually Work

"How does an LLM generate a response?"

This is the question you will be asked in every AI / Applied Engineer interview. Practice until you can deliver it in 60–90 seconds without sounding like you're reciting it.

Miss one and they probe.

1. Tokens — sub-word units; the model reads these, not words
2. Embeddings — each token becomes a vector of numbers capturing its meaning
3. Attention — scores relevance across every token in context, for each new token generated
4. Generation / sampling — next-token prediction, one at a time, until done
5. Hallucination root cause — plausibility ≠ truth; there is no truth-check step in the loop

"An LLM converts your input into tokens — sub-word units — then maps each token to an embedding vector that captures meaning. An attention mechanism scores which parts of the context are relevant when generating each new token. The model then repeatedly samples the next most probable token from a probability distribution until done. It is not retrieving facts — it is predicting statistically plausible continuations — which is why it can sound confident while being wrong, and why anything it doesn't know must be explicitly provided in context."

Your version doesn't have to match word-for-word — the five beats must all be present.

• "How does an LLM generate a response?" — asked in every AI engineering interview
• "Why did it hallucinate?" — near-guaranteed follow-up
• "What's the context window?" — system design interviews
• Naming the mechanism separates senior from junior answers

• Chatbot forgetting earlier messages → context window constraint
• API costs spiking with long inputs → token pricing
• Model confidently giving wrong answers → hallucination root cause
• Poor output from cluttered prompts → attention and noise

An LLM converts your text into tokens — sub-word fragments — and maps each to an embedding: numbers capturing meaning. An attention mechanism scores which tokens matter for each generation step. The model then produces output by repeatedly sampling the next most probable token — one at a time, until done.

This is not retrieval. There is no fact-checking step — confident tone is a learned style, not an accuracy signal.

The context window is a fixed-size buffer that resets on every request — not persistent memory. It's the central engineering constraint: every design decision in Sessions 2 through 4 is about what to put in that buffer, and in what order.

The smallest chunk of text the model reads — a sub-word fragment, not a word, not a character. "unbelievable" = 3 tokens; "ChatGPT" = 3 tokens.

Like the syllables a speed reader sees rather than whole words. You pay per token, not per word.

Context is everything the model can see in one conversation — your messages, documents, its own replies. Like working memory, not long-term memory.

Context window is the size limit on that context — the max tokens (input + output) per request. Resets completely after every call, like a whiteboard erased after each meeting.

A list of numbers representing what a token means and how it's used — not its spelling.

Like GPS coordinates for meaning: "fast" and "quick" sit nearly on top of each other; "fast" and "banana" are continents apart.

The mechanism that decides which tokens in context are relevant to the next token being generated — computed fresh for every word, on every step.

Explains why cluttered prompts produce worse output: noise competes for attention with your actual instruction.

The model's core operation: given everything in context, score every possible next token, sample one, append it, repeat.

No retrieval. No truth check. Pure statistical prediction, one token at a time, until done.

Controls how conservative or adventurous the model is when picking the next token.

Low (0.0–0.3): consistent, predictable — fact extraction, structured output.
High (0.7–1.0): varied, creative — brainstorming.

When the model states something confidently and authoritatively that is factually wrong or made up.

Not a bug — a direct consequence of optimizing for plausibility without a truth-check step. Confident tone is a learned style, not an accuracy signal.

The date after which the model has no built-in knowledge. Anything more recent — or any private data — must be supplied in the context window.

This is the entire premise of RAG (Session 3).

Self-assessment, not graded. Answer before checking your notes.

1. An LLM generates text by:

• Looking up the most relevant pre-written answer in a database
• Predicting the most statistically likely next token, one at a time
• Running a search engine query and summarising the top results
• Executing a fixed set of grammar and reasoning rules

2. The context window is best described as:

• The model's persistent long-term memory of past conversations
• The maximum tokens (input + output combined) the model can process in one request
• The time period during which the model's training data was collected
• The number of users who can query the model simultaneously

Self-assessment, not graded.

3. In 2–3 sentences: why can an LLM state a false fact with high apparent confidence?

4. Embeddings are best described as:

• Compressed copies of the original text stored for fast retrieval
• Numeric vectors representing meaning, where similar meanings map to nearby points
• A sorted list of every token in the model's vocabulary
• The model's confidence score for each token it outputs

Self-assessment, not graded.

5. Which is NOT a direct consequence of LLM architecture covered this session?

• Cost scales with the number of input and output tokens
• The model cannot know about events after its training cutoff unless told
• The model can perfectly retrieve the exact wording of any training sentence
• Very long, unfocused prompts can dilute output quality

6. Interview-style: "Why would you add a retrieval step (RAG) instead of asking the model directly?" — answer in 2 sentences using only this session's concepts.

Self-assessment, not graded.

7. What does temperature control?

• The maximum number of tokens the model will generate
• How peaked or flat the next-token probability distribution is
• The speed at which the model processes the context window
• Whether the model searches external sources before responding

8. Give one concrete hallucination example you might hit building a product feature, and why it's dangerous in production.

≈45 min · connects directly to cost/system-design interview questions

Goal: connect token counts to actual engineering cost before you build anything.

Take any paragraph you'd send as a system prompt on every user query. Count its tokens (any tokenizer playground, e.g. tiktokenizer.vercel.app). Assume 1,000 queries/day.

Write down: your paragraph's token count, its daily cost at two different models you'd realistically consider, and one design change that would halve the cost without removing the feature.

≈45 min · cements the "resets every request" concept from Deep Dive

Goal: observe what actually happens when a request approaches the context window, then draw a design conclusion from it — not just memorize the definition.

Find a long technical document (5,000+ words). Paste it into a free LLM chat interface, or use API access with a deliberately small max_tokens input limit. Ask a question about content near the beginning, then near the end.

Write down: which parts the model could and couldn't access, and two sentences on what this forces a production system that processes long documents to do differently.