Model Selection & Fine-Tuning Trade-offs

"Walk me through how you would choose a model for a production AI feature. What factors would you consider, and when would fine-tuning be the right call?"

Model selection is the first architectural decision in any AI feature — and one of the most commonly botched. Deliver the full framework in 90 seconds.

Missing beat 1 signals shallow thinking; missing beat 5 signals over-reliance on public benchmarks.

1. Name the constraint first — latency, cost, or quality, before any model name
2. Match tier to constraint — frontier / mid-range / lightweight follows from the constraint, not brand preference
3. Exhaust prompting and RAG first — both faster to iterate, easier to reverse than fine-tuning
4. Fine-tune only with all 4 signals present — style drift, rare vocabulary, tight latency budget, very high volume
5. Validate with your own evals — leaderboard scores tell you where to start, not where to stop

"The first question I ask is: what's the binding constraint — latency, cost, or quality? That collapses the decision space immediately. For latency-sensitive tasks I start mid-range; for high-stakes reasoning with no latency pressure, frontier; for high-volume simple classification, lightweight — the tier follows from the constraint, not brand preference. Before any fine-tuning conversation, I invest seriously in prompt engineering — at least ten meaningfully different strategies — and RAG if the problem is factual accuracy, since both are faster to iterate and easier to reverse. I only evaluate fine-tuning if I have consistent style drift prompting can't hold, rare domain vocabulary, a latency budget a long system prompt can't meet, or very high volume on a narrow task — all four should apply. The final step is always running evals on my own data and my own task — a model that scores second on a general benchmark may outperform the leader on my specific domain."

• "How would you choose a model for a production feature?" — the standard system-design opener for any AI role
• "When would you fine-tune instead of prompting?" — near-guaranteed follow-up that separates real experience from theory
• "Why not just use the biggest, best model everywhere?" — tests whether you understand the cost-latency-quality trade-off
• Citing your own evals over leaderboard scores is what a senior engineer sounds like

• A feature is slow and expensive → frontier model used where lightweight would meet the accuracy bar
• Quality got worse after upgrading to a bigger context window → attention dilution, not a broken model
• Team jumps straight to fine-tuning for a formatting quirk → prompting and RAG never seriously exhausted first
• A provider update silently changes output format in production → model version wasn't pinned

Every model choice is governed by the cost-latency-quality triangle: improving one dimension tends to worsen at least one other. You name the binding constraint for your use case first, then pick the model tier (frontier, mid-range, lightweight) that matches it — never the reverse.

Fine-tuning and RAG solve different problems: RAG updates knowledge instantly at inference time; fine-tuning bakes it into the weights and is hard to reverse.

Every other beat in this session follows from this one. Skipping straight to a model name — "we should use the frontier model" — is the single most common signal of shallow thinking in a model-selection interview answer. The constraint (latency, cost, or quality) determines the tier; the tier determines the model.

A grouping of AI models by capability level, cost, and intended use case — frontier, mid-range, or lightweight. You choose a tier based on your constraints, then select a specific model within it.

Like choosing a car class (economy, sedan, luxury) before picking the exact make and model.

The one dimension — latency, cost, or quality — that cannot flex for your use case. The first thing to name before any model selection discussion.

The one non-negotiable requirement in a job posting — everything else is a nice-to-have by comparison.

The three-way trade-off in model selection: improving any one dimension tends to worsen at least one other. You optimise for two; the third flexes.

Fast, cheap, good — pick two. No model gives you all three at once.

The phenomenon where a model's attention spreads thin over a very long context, causing content in the middle of the window to receive less attention and potentially be "forgotten" even when technically present.

A crowded lecture hall — the people in the middle rows get noticed less than the front or back, even though everyone's technically in the room.

Continuing to train a pre-trained model on a smaller, domain-specific dataset — adjusting the model's weights to improve performance on a specific task or style. Changes weights; hard to reverse.

Sending someone to a specialised trade school — they come back genuinely different, but you can't easily un-train them.

RAG retrieves relevant documents and injects them into context at inference time — updates are immediate, no retraining needed. Fine-tuning bakes knowledge into the weights — updating it requires retraining.

RAG is an open book you can restock overnight; fine-tuning is what the person actually memorised.

When training data contains examples from a benchmark's test set — accidentally or intentionally — inflating the model's score on that benchmark beyond its true capability.

Studying with the actual exam questions beforehand — the score stops measuring what it claims to measure.

A single module or interface in your codebase that handles all LLM API calls — so switching models or providers requires changing configuration, not rewriting business logic.

A universal power adapter — build it once, and swapping what's plugged in doesn't mean rewiring the house.

Self-assessment, not graded. Answer before checking your notes.

1. An AI feature must process 200K daily active users with responses expected in under 1.5 seconds. What is the binding constraint?

• Quality — user trust depends on answer accuracy
• Latency — users expect fast responses
• Cost — 200K users means high token volume
• Context window — many users require long conversations

2. Your team has tried eight different system prompt strategies over three weeks and still can't hit the required style consistency. What should you try next?

• Fine-tune the model on your examples immediately
• Upgrade to a frontier model tier
• Try RAG to inject style examples at inference time
• Reduce the output length requirement

Self-assessment, not graded.

3. Which benchmark would most reliably predict a model's performance extracting named entities from legal contract clauses?

• MMLU (general knowledge breadth)
• HumanEval (Python code generation)
• MATH (mathematical reasoning)
• None of the above — your own eval on your task

4. A developer upgraded to a 200K token context model and quality on their Q&A task got worse. Most likely explanation?

• The model's weights degraded over time
• Attention dilution — critical content in the middle of a large context is under-attended
• The context window is too small for the task
• The prompt was accidentally shortened during the upgrade

Self-assessment, not graded.

5. Which of the following is a signal that fine-tuning is the right approach?

• The model occasionally gives slightly different formatting in the response
• You need factual accuracy on a 500-document internal knowledge base
• You're running 60,000 classification calls/day and a lightweight model with prompting can't hold 93% accuracy
• The latency is 800ms and you'd prefer 500ms

6. You're designing a customer support system. Which approach correctly applies the cost-latency-quality triangle?

• Use a frontier model for all calls — quality is non-negotiable
• Use a lightweight model for simple intent routing and a frontier model for complex resolution
• Use the same model for all calls to avoid architectural complexity
• Use a frontier model during business hours and a lightweight model overnight

Self-assessment, not graded.

7. A model's benchmark score dropped three months after release with no model change. Most likely explanation?

• The model's weights degraded over time
• The evaluation methodology was updated to remove contaminated test examples
• The model was secretly downgraded by the provider
• Other models improved, so the relative score dropped

8. Interview-style: what is the primary difference between RAG and fine-tuning for improving factual accuracy in a domain-specific Q&A system, and when would you choose one over the other?

≈1.5 hr · builds the evidence for Interview Beat 5

Goal: stop trusting leaderboard scores and generate your own comparison data across model tiers.

Write a script that runs the same 20 fixed queries through three model tiers (lightweight / mid-range / frontier) and records, per call: latency (ms), input tokens, output tokens, and a response preview. Save results to benchmark_results.json.

Write down: which tier had the widest latency spread, and one query where a cheaper tier's answer looked just as good as the frontier tier's.

≈1 hr · turns raw numbers into a defensible recommendation

Goal: convert your benchmark data into the kind of evidence you'd actually cite in a system design interview.

Manually score each response 1–5 for quality (5 = complete and correct, 1 = incorrect or hallucinated). Aggregate into a table: avg latency, avg input/output tokens, estimated cost per 1K calls, avg quality score, and % scoring ≥4 — one column per tier.

Write down: based on your table, which tier you'd recommend for a consumer feature with 5,000 daily users needing sub-3-second responses, and why.