GenAI engineer interviews mostly ask you to make and defend trade-offs, not to define terms. Expect questions on how a language model works, how you would build retrieval, how you stop an agent from doing damage, how you measure quality, and how you pick a model. A good answer states a choice, gives the reason, names what it costs, and says how you would check it worked.
Below are the question groups that come up most, with a short model answer for each. Use them to hear the shape of a good answer, not to memorize wording. Interviewers can tell a recited answer from a reasoned one within two follow-ups.
What does a good answer sound like in general?
Almost every strong answer has four parts:
- The decision. “I would use retrieval here, not fine-tuning.”
- The reason. “The facts change weekly, and a trained-in fact goes stale.”
- The cost. “Retrieval adds a search step, so latency goes up and I now have a second system that can fail.”
- The check. “I would measure whether the right passage is in the top five results before I touch the prompt.”
A weak answer stops after part one, or recites a definition. If you have only ever heard definition-style questions, read why AI engineering interviews stopped asking you to define things first.
How do LLMs work, and why does it matter for the job?
Typical question: “Why did the model give a different answer to the same prompt?” or “Why does it cut off long documents?”
What a good answer sounds like: The model reads tokens, not words, and predicts the next token from a probability spread. Settings such as temperature change how much it samples from that spread, so identical prompts can differ. Input and output share one limited context window, so a long document plus a long answer can run out of room. A strong candidate adds the practical consequence: count tokens before sending, and set the temperature low for tasks that need repeatable output.
What gives a weak answer away: saying “it understands” or “it reasons like a person”, with no mention of tokens or limits.
The first chapter of the course covers this ground: how LLMs actually work.
How do you write prompts that hold up in production?
Typical question: “Your prompt works on your five test cases and fails for users. What do you do?”
What a good answer sounds like: Five cases is not a test set. Collect real failures, group them by cause, and write a fixed set of inputs with expected outputs. Change one thing at a time, run the whole set, and compare the pass rate before and after. Ask for a fixed output format so that code can parse the reply, and handle the case where the format is wrong anyway.
A senior touch: say that a prompt change that fixes one case often breaks another, which is why you keep the set and rerun it every time.
See how to tell a prompt change actually helped for a worked version of this.
How would you build retrieval-augmented generation?
Typical question: “Design a question-answering system over our internal documents.”
What a good answer sounds like: Split documents into chunks, turn each chunk into an embedding (a list of numbers that places similar meaning close together), store them, and at question time fetch the nearest chunks and put them in the prompt. Then talk about where it breaks:
- Chunks that are too small lose the context around a fact. Chunks that are too large bury it.
- The right passage may exist but not be retrieved.
- The right passage may be retrieved but the model ignores it.
Those are two different failures and they need two different measurements. Here is a small way to measure the first one:
def recall_at_k(results, gold_chunk_id, k=5):
"""1 if the known-correct chunk is in the top k results, else 0."""
return int(gold_chunk_id in results[:k])
# Average this over 50 real questions that each have a known answer chunk.
If recall is low, fix chunking or search. If recall is high and answers are still wrong, fix the prompt or the model. A candidate who says this has shown they debug in layers instead of guessing.
Related reading: understanding RAG: when to use it and when not to and the chapter on embeddings and RAG.
RAG, fine-tuning or a better prompt: how do you choose?
Typical question: “The model doesn’t know our product. Should we fine-tune it?”
What a good answer sounds like: Start with the cheapest lever. If the model lacks facts, give it the facts at question time with retrieval. If it knows the facts but answers in the wrong style or format, try a clearer prompt with examples first. Consider fine-tuning when you have a steady, narrow task, plenty of good example pairs, and a prompt that is already as good as you can make it. Fine-tuning changes behaviour more reliably than it adds new facts, and it needs a retrain whenever the facts change.
A table helps in the room:
| Problem | First thing to try | Why |
|---|---|---|
| Model lacks current facts | Retrieval | Facts can be updated without retraining |
| Wrong tone or format | Prompt with examples | Cheap to change and test |
| Narrow task, large volume, cost matters | Fine-tune a smaller model | Can cut per-call cost once the task is stable |
More on this: fine-tuning vs prompt engineering.
How do agents and tool calling work, and how do you keep them safe?
Typical question: “Your agent can send email and read the database. What could go wrong?”
What a good answer sounds like: The model never runs anything. It asks for a tool call, and your code decides whether to run it. That makes your code the place for permissions. Give each tool the least access it needs, validate arguments before running, require a human to approve actions that cannot be undone, and cap the number of loop steps so a confused agent stops instead of looping. Mention that text inside a fetched web page or document is untrusted and can try to instruct the model.
What gives a weak answer away: describing the happy path only, with no limits, no logging and no approval step.
Go deeper with the permission model hiding inside every AI agent and the chapter on tool use and single-agent loops.
How do you know your GenAI system is good?
Typical question: “How would you evaluate this before launch?”
What a good answer sounds like: Build an evaluation set from real or realistic questions with known good answers. Score what can be scored by code (format valid, correct source cited, right tool chosen). For open-ended quality, use a second model as a judge, but check the judge against a sample of human grades, because a judge can be wrong in a consistent way. Track the numbers over time and rerun them on every change to the prompt, model or retrieval. After launch, log inputs and outputs so you can find new failure types.
The key sentence interviewers listen for: “I would decide what good means before I build, and I would be able to show a number.”
How do you pick a model?
Typical question: “Which model would you use for this feature?”
What a good answer sounds like: Do not name a favourite. Say that you would try a small, fast model first on your evaluation set, move up only when it fails on cases that matter, and weigh three things together: answer quality, response time and cost per call. A cheap model for easy steps and a stronger one for hard steps is common. Add that prices and model lineups change often, so you would check the vendor’s current pricing page and rerun the evaluation when a new model arrives.
See how to choose an LLM model tier.
How would you design this for production?
Typical question: “Take your prototype to 10,000 users. What changes?”
What a good answer sounds like: Walk through the layers in order: where requests enter, what you cache, how you handle rate limits and failures (retry with backoff, fall back to a simpler path), how you control cost per user, what you log, how you monitor quality drift, and how you roll out a change safely to a small share of traffic first. Name one thing you would deliberately not build yet.
This is the system-design round, and it is where most candidates have the least practice. The chapter on production AI system design walks through it in the order interviewers tend to probe.
What should you do this week?
- Pick three questions from the groups above and answer each out loud in two minutes, using the four-part shape: decision, reason, cost, check.
- Find the one you stumbled on and read the matching chapter.
- Check where you stand with the AI/ML readiness quiz.
- See what the AI interview page offers for practice.
- Finish with the free chapter on production AI system design, since that round combines everything above.
If you can say what you chose, why, what it costs and how you would check it, you will sound like someone who has built the thing, which is what the interviewer is trying to find out.