Production AI System Design

"Walk me through how you'd take an AI feature from prototype to production."

The core system-design question for any AI/Applied Engineer role. Practice until you can deliver all five beats in under two minutes without listing tools — the interviewer is grading judgment, not vocabulary.

Miss one and they probe.

1. Define the system — real load numbers and what "quality failure" means for this feature
2. Stack the five layers — gateway → orchestration → LLM provider → cache/storage → observability
3. Name the tradeoff — where you sit on the latency/cost/quality triangle, and why
4. Reliability patterns — retry, timeout, circuit breaker, fallback, in that order
5. Observability — the specific metrics you'd alert on, not just "we'd monitor it"

"This feature handles 500 users/day, roughly 25K requests/month — quality bar is no hallucinated sources, so we need an eval feedback loop before launch. Stack: API gateway for auth/rate-limiting → orchestration layer for retries and tool calls → Anthropic as the LLM provider → Redis for exact-match caching plus pgvector for semantic search → Langfuse for observability. We sit closer to quality than cost, since wrong sources destroy trust — managed via model routing, small model for simple queries, larger model only for heavy tool-calling chains. Reliability: three retries with exponential backoff, a 10-second timeout, a circuit breaker that opens after five consecutive failures with a 30-second cooldown, and a static fallback message. We'd monitor P99 latency, error rate, cost per request, automated eval score, and cache hit rate — alerting the moment any of them drifts from baseline."

Swap in your own numbers and stack — the five beats, not the exact tools, are what's being graded.

• "How would you productionize this?" — the standard closing system-design question in AI/ML interviews
• "What happens when the API goes down?" — near-guaranteed follow-up if you don't cover reliability upfront
• "How would you know if quality degraded?" — probes whether you understand AI-specific observability, not just infra metrics
• Naming the five-layer architecture by name separates senior from junior answers

• Feature works in testing, then latency spikes to 45 seconds in production → missing timeout, unscoped for real traffic
• API bill triples in a month with no usage change → prompt bloat / context drift
• Users get wrong answers but no errors show up in logs → silent quality degradation, needs an eval score, not just infra metrics
• One slow LLM provider call backs up the entire queue → no circuit breaker

A prototype only has to work once, in a demo, with clean input. A production system has to handle unpredictable load, degrade gracefully under failure, cost a predictable amount, and stay monitorable when something breaks — for every request, not just the happy path.

This gap — between "it works on my machine" and "it works for 500 real users at 2am" — is exactly what this session's five layers and four reliability patterns are built to close.

Every AI system sits on a latency / cost / quality triangle — you can optimize for any two, never all three. Fast + cheap sacrifices quality (autocomplete). Fast + good sacrifices cost (customer-facing answers). Cheap + good sacrifices speed (batch analytics).

Every other design decision in this session — model routing, caching, which reliability pattern to add first — is downstream of where you deliberately choose to sit on this triangle.

Every production AI request passes through: 1. Gateway (auth, rate-limiting) → 2. Orchestration (builds prompts, manages retries, calls the LLM) → 3. LLM Provider (runs the model) → 4. Cache & Storage (prompt/response pairs, retrieval data) → 5. Observability (logs, metrics, traces, alerts).

Like a relay race — every request runs the full course, and a failure at any single leg can sink the whole request.

The three-way tradeoff governing every AI system design decision. You can pick any two corners; the third is what you give up.

Like ordering food: fast and cheap, fast and good, or cheap and good — never all three at once from the same kitchen.

When a request fails transiently (e.g. HTTP 429), retry after a delay that doubles each time — 1s, 2s, 4s — rather than immediately.

Like waiting longer between each knock on a door that isn't answering, instead of hammering it — giving the other side room to recover instead of adding to the problem.

After a threshold of consecutive failures, stop sending requests entirely for a cooldown period, then let a small fraction through (half-open) to test recovery before fully resuming.

Like a household circuit breaker tripping to stop a surge from burning down the wiring — better to stop the flow than let it burn out the whole system.

When retries and the circuit breaker have both given up, return something useful anyway — a cached answer, a simpler model's response, or a static message — instead of a blank error.

The difference between a store saying "out of stock, try this similar item" versus just locking the doors.

Send each request to the model tier that matches its complexity — small/fast models for simple queries, large/capable models only for heavy reasoning or tool-calling chains.

Cuts average cost per request 60–80% versus routing everything to the most expensive model — like not hiring a specialist surgeon to check a routine blood-pressure reading.

The latency value that 99% of requests beat. The slowest 1% is your worst realistic user experience — not an edge case to ignore.

Average latency is the weather forecast; P99 is what actually happens to the unlucky commuter caught in the one bad storm — at 10,000 users/day, that's 100 people a day having a bad time.

A gradual, often invisible increase in prompt token count over time — untrimmed conversation history, or a RAG pipeline injecting more retrieved content with each call.

Like a desk that slowly disappears under papers — nothing changed today, but the accumulation degrades focus, raises cost, and dilutes attention on what actually matters.

Self-assessment, not graded. Answer before checking your notes.

1. A customer-facing AI chat feature gets a 429 (rate-limited) error from the LLM API. What's the correct sequence of responses?

• Return an error immediately to the user
• Retry once after 1 second; if it fails again, return an error
• Retry with exponential backoff (1s, 2s, 4s); if all three fail, return a cached or static fallback
• Retry indefinitely until the API responds

2. A summarisation feature tested well on 10 clean documents. On launch, latency spikes to 45 seconds and some documents return garbage. What are the two most likely root causes?

• The model is too small for the task / the users are sending too many requests
• The prompts were designed for clean data, and there's no timeout set

Self-assessment, not graded.

3. Which three metrics would you check first if quality degraded silently — wrong answers, but no HTTP errors?

• P99 latency, cache hit rate, error rate
• Automated eval score, token usage, user feedback signal
• Cost per request, retry rate, circuit breaker trip count

4. Three request types: simple yes/no classification, 500-word summarisation, multi-step research with tool calls. What's the correct model-routing strategy?

• Same model for all three, for consistency
• Route by complexity: small model for classification, medium for summarisation, large/capable for the tool-calling chain

Self-assessment, not graded.

5. A circuit breaker opens after 5 consecutive failures and has been open 30 seconds. What happens next?

• Stays open permanently until manually reset
• Closes fully and resumes all traffic immediately
• Enters a "half-open" state, letting a small percentage of traffic through to test recovery

6. Which is NOT a valid use case for LLM response caching?

• A personalised birthday message for a specific named friend
• "What are your opening hours?" across many users
• Semantically similar FAQ questions with the same correct answer

Self-assessment, not graded.

7. Interview-style: an interviewer asks you to walk through taking a feature from prototype to production. In 2-3 sentences, what are the five things your answer must cover?

8. A feature keeps returning HTTP 200 but quality degrades over two weeks, with token usage trending up 10%/day. What's the most likely cause, and what would you check to confirm it?

≈45 min · forces you to commit to a concrete stack, not just recite layer names

Goal: turn the five abstract layers into a specific system you could actually defend in an interview.

Pick any AI feature (a project from an earlier session works well). Draw all five layers — Gateway, Orchestration, LLM Provider, Cache & Storage, Observability — labelled with the actual tool/service you'd use at each. Show the request flow end to end, and separately show the failure flow: what happens when the LLM returns a 429 — which layer catches it, and what the user sees.

Write down: your five layers with tools named, and one sentence on which layer you're least confident about and why.

≈1 hr · the exact artifact interviewers ask you to talk through

Goal: commit to a tradeoff position and a reliability configuration in writing, not just in conversation.

For the same feature from Assignment 1, write: which corner of the latency/cost/quality triangle you prioritize and why; a model-routing table (query type → model tier → reasoning); and a reliability table (retry / timeout / circuit breaker / fallback → your configuration for each).

Write down: your triangle position in one sentence, and the exact configuration values (retry count/delays, timeout seconds, circuit-breaker threshold/cooldown) you chose.