Applications of Foundation Models

Chapter 08 — Designing FM Applications: Selection, Inference Parameters, and Agents

Certification Blueprint

Field	Coverage
Exam	AWS Certified AI Practitioner (AIF-C01), exam guide v1.1
Domain	Content Domain 3 — Applications of Foundation Models (28% of scored content)
Task statement	Task 3.1 — Describe design considerations for applications that use foundation models
Objectives	3.1.1 selection criteria · 3.1.2 inference parameters · 3.1.6 the role of AI agents
Service families	Foundation model providers and their inference APIs; agent runtimes

What This Chapter Covers

Domain 2 asked whether generative AI belonged in a problem at all, and which broad family of model suited the shape of the output. This chapter assumes those questions are answered and asks the next three: which specific model, set up which way, doing what kind of work.

Domain 3 is the heaviest domain on the exam at 28% of scored content, and Task 3.1 is its widest task statement with six objectives. Three of them are here. The other three — retrieval augmented generation, vector stores, and the customization cost ladder — are Chapter 09.

The character of the questions changes in this domain, and it is worth knowing before you start. Domain 1 and Domain 2 questions could often be answered from principle. Domain 3 questions describe a scenario in which almost everything is satisfactory, and the work is finding the one thing that is not. Reading for the most thorough-sounding option is the reliable way to lose marks here.

The Core Mental Model

Everything in this chapter is one screen with two halves. First you eliminate candidate models on the properties they either have or do not have. Then you rank whatever survives on what it costs you.

If that shape looks familiar, it should — Chapter 07's eight-factor screen has the same structure. What changes between the two chapters is not the method but which properties sit in the gates, and recognising the repeated shape is worth more than memorising either list in isolation.

The Nine Selection Criteria

Objective 3.1.1 names nine criteria for choosing a foundation model. The examinable skill is not reciting them — it is knowing which ones eliminate a candidate outright and which merely rank it against the others.

Criterion	The question it asks	Behaves as
Modality	Does it accept what we send and return what we need?	Eliminates
Multi-lingual	Does it serve every language our users write in?	Eliminates
Input/output length	Are its maximums large enough for our longest case?	Eliminates
Latency	Can it answer inside the time budget?	Eliminates
Customization	Can it be adapted the way this use case needs?	Eliminates or ranks
Prompt caching	Can it reuse a repeated prefix instead of re-charging it?	Ranks
Model size	How much model are we paying to run?	Ranks
Model complexity	How much capability does the job actually need?	Ranks
Cost	What does it cost at the volume we expect?	Ranks

Customization is the one genuinely ambiguous row, and the ambiguity is real rather than a gap in these notes. If a use case requires an adaptation a model cannot support at all, customization eliminates. If several candidates all support it at different costs, it ranks. An exam scenario will make clear which situation you are in.

A criterion that every candidate satisfies has not helped you. Scenarios are built so that most criteria are satisfied and one is not; that one is the answer.

These are not Chapter 07's eight factors

Two objectives in this exam name selection criteria, and they are different lists for different decisions. Confusing them is the most reliable single way to lose a Task 3.1 question.

	Objective 2.2.3 (Chapter 07)	Objective 3.1.1 (this chapter)
Decides	Whether generative AI fits, and the shortlist	Which model, given the shortlist
Count	Eight factors	Nine criteria
Appears in both	Cost, latency, model complexity	Cost, latency, model complexity
Only in that list	Model types, performance requirements, capabilities, constraints, compliance	—
Only in this list	—	Modality, multi-lingual, model size, customization, input/output length, prompt caching

Compliance and constraints are not on the Objective 3.1.1 list. They are real considerations and they matter enormously in practice — they simply belong to the earlier decision. By the time you are separating one foundation model from another, the shortlist you are working from is already a compliant one. An option that offers compliance as the deciding criterion in a Task 3.1 question is usually offering a Domain 2 answer to a Domain 3 question.

The three criteria that appear in both lists — cost, latency, model complexity — are why this confusion is so easy to fall into. A candidate quoting the wrong list is often accidentally right, which is exactly what stops them noticing.

Modality is a pairing, not a label

A model described as multi-modal may accept images and return only text, or generate images from a text prompt and accept nothing but text as input. "Supports images" does not tell you which direction. An application that reads scanned invoices needs image in; an application producing marketing artwork needs image out. A model that does the second and not the first is genuinely multi-modal and completely useless for the first job.

Always name both halves of the pairing you need before you screen anything.

Prompt caching

Prompt caching means reusing the already-processed form of a repeated prompt prefix instead of paying to process it again on every request.

The shape it suits is specific, and the exam describes the shape rather than naming the feature: a large fixed prefix — a system instruction, a policy document, a schema, a set of examples — followed by a short variable part, at meaningful volume. If an application sends the same 4,000 tokens on every one of forty thousand daily requests, that prefix is being processed and charged forty thousand times, and caching is the criterion that addresses it.

The common wrong instinct is to reach for a smaller model instead. That reduces the unit price and still pays for every token, every time. Caching removes the repeated work; a smaller model discounts it. The two levers act on different halves of the same bill.

Input/output length means two different things

The same words appear in Objective 3.1.1 and in Objective 3.1.2. This is not a duplication — it is two different decisions that happen to concern the same quantity.

In Objective 3.1.1 (a selection criterion)	In Objective 3.1.2 (an inference parameter)
The model's maximum context and response size	The value you set on a given request
You shop for it; you cannot change it	You choose it, inside that maximum
Getting it wrong means the model cannot do the job at all	Getting it wrong means the response is cut off

The test to run under exam pressure: can I change this without changing model? If not, you are looking at the selection criterion. If yes, you are looking at the parameter.

Inference Parameters

Inference parameters are settings supplied with a request that change how the model produces its response — without changing the model, the prompt, or what the model knows. Objective 3.1.2 names temperature and input/output length. In practice you will also meet top-p and top-k, which are adjacent sampling controls the objective does not name.

Two ways to hold the idea:

• The settings on an oven, not the recipe. Temperature and timer change how the dish comes out. Neither changes what went into the tin, and neither will rescue the wrong ingredients.
• Asking a colleague for "one option" against "a few options, roughly sketched." Same person, same knowledge, same question — a different instruction about how widely to range.

Temperature

A model produces a distribution over possible next words. Temperature controls how widely it samples from that distribution.

Setting	What you get	Suits
Low	The most likely wording, almost every time	Extraction, classification, structured output
High	Less likely wordings appear more often	Drafting, ideation, generating options to choose from

Temperature does change	Temperature does not change
How much the wording varies between runs	Whether the answer is factually right
How reproducible a pipeline looks	What the model knows
Whether you get one answer or a range	Whether the source material was any good

A low temperature does not make a model correct. It makes it consistent. If the model is wrong about something, a temperature of zero produces the same wrong answer every time, with complete confidence. That is worse than an intermittent error, not better, because it stops looking like a glitch and starts looking like a rule.

Chapter 07 established that nondeterminism is inherent and cannot be configured away. This chapter adds the other half: reducing variation is not the same as adding accuracy, and reaching for temperature when the problem is accuracy is moving along the wrong axis entirely.

Reproducibility is still genuinely valuable — it makes a pipeline testable, debuggable and auditable. It is simply a different thing from correctness, and the exam tests whether you know that.

Output length is a budget, not an instruction

The maximum output length is a hard stop on generation. When the model reaches it, generation ends — wherever it happens to be.

What teams expect	What actually happens
"Setting it low makes the model concise"	The model writes at its natural length and is cut off mid-sentence
"It will summarise to fit"	Nothing summarises; the ceiling truncates
"It saves money safely"	It saves money and silently damages the output

If you want brevity, ask for it in the prompt — that is an instruction, and writing instructions is Chapter 10's material. The ceiling is a safety limit against runaway generation and a cost control.

The symptom tells you which one you hit. A model that was asked to be brief produces something short and complete. A model that hit its ceiling stops mid-sentence.

Three settings, three different failure modes — and only one of them announces itself. Exceeding the input limit produces an error you cannot miss. A badly chosen temperature produces output that looks fine until you compare runs. An output ceiling produces a truncation only a reader notices.

The Role of AI Agents

An AI agent is the component that turns a stated goal into a sequence of actions against real systems — planning a step, taking it, reading the result, and deciding the next one — until the goal is met or a stop condition is reached.

Two ways to hold it:

• A travel agent, not a timetable. A timetable answers questions about trains. An agent books the trip, and rebooks it when the first option sells out.
• A new colleague given an outcome rather than a checklist. "Get this invoice paid" rather than "open the portal, click approve." They work out the steps, and they come back when it is done or when they are stuck.

The sentence to carry into the exam: a chatbot answers a question; an agent completes a task.

Chapter 06 asked what an agent is. This objective asks what it is for

	Objective 2.1.6 — Domain 2 (Chapter 06)	Objective 3.1.6 — Domain 3 (here)
Asks	What is an agent made of?	What is an agent for?
Covers	Multi-agent patterns, the Model Context Protocol, memory management, tool usage, workflow orchestration	The role an agent plays in an application, and the business applications it serves
A good answer names	Components	Outcomes

⚠️ The Model Context Protocol appears nowhere on the Domain 3 page. In the v1.0 exam guide it sat inside Objective 3.1.6 as a parenthetical example; v1.1 removed it from there and promoted it to its own objective, 2.1.6, in Domain 2. Anyone revising from a v1.0 guide — or from a summary written from one — will answer a Domain 3 agent question with Domain 2 material, confidently. Check the live page yourself before the exam; this is a fact with a shelf life.

Business applications

Objective 3.1.6 asks for business applications, and the pattern repeats in every one: a task that used to end with a human doing the steps.

Application	What the chatbot version did	What the agent version does
Customer resolution	Explained how to request a refund	Issues the refund, updates the order, sends confirmation
IT and operations triage	Suggested likely causes of an alert	Queries the systems, correlates, opens or closes the incident
Research and reporting	Answered questions about sources one at a time	Gathers across sources and assembles the draft report
Back-office workflow	Told staff which form was needed	Completes the form, files it, records the exception
Software development	Explained how to fix the failing test	Edits the code, runs the test, iterates until it passes

The value claim is identical in every row: the handoff to a human disappears. So does the review step, unless it is deliberately designed back in. An agent removes a human checkpoint by default, and putting one back is a design decision somebody has to make — which is where Domain 4's responsible-AI material connects to this one.

The loop is what makes it an agent, and the loop is what makes it expensive — every pass is another model call. The stop condition is the part that gets designed last and matters most: without one, an agent that cannot achieve its goal keeps trying.

When an agent is the wrong answer

An agent buys exactly one thing: the sequence of steps does not have to be known in advance.

The process	Better fit	Why
Steps vary by case; you cannot enumerate them ahead of time	Agent	Planning per case is the capability being bought
Same steps, same order, every time	Fixed workflow	The planning loop adds cost, latency and variability, and solves nothing
Steps known, but which run depends on data	Workflow with branches	Conditional logic is cheaper and auditable
One question, one answer, no action	Plain model call	There is no task to complete

What the loop costs: several model calls per task instead of one, latency compounding across them, and a system that now acts — so a wrong step changes real data rather than producing a wrong sentence.

This is Chapter 02's lesson two levels down. That chapter said sometimes the answer is not to use AI. Chapter 07 said sometimes the answer is to reject generative AI for a requirement. This chapter says sometimes the answer is a workflow. The option recommending the simpler thing is frequently correct and frequently looks least impressive.

Decision Rules and Exam Signals

When the question says…	Reach for	Because
Users write in several languages	Multi-lingual, eliminating	A language the model cannot serve is a failure, not a lower score
Scanned documents, images or audio arrive	Modality, eliminating	Name both halves of the pairing before screening anything
The same large instruction block on every request	Prompt caching, ranking	The cost is in repeated processing, not unit price
"Our longest contract fails"	Input/output length as a maximum	The criterion was tested against the typical case
"Set max tokens lower to save money"	Output ceiling, and a warning	It truncates; it does not summarise
"Lower the temperature until the errors stop"	Temperature, and a correction	It changes spread, not correctness
One correct output exists	Low temperature	Variety is a defect, not a feature
Several drafts for a human to choose from	Higher temperature	Variety is the deliverable
The steps vary case by case	An agent is warranted	The unknown sequence is what you are buying
The same steps every time, in order	A fixed workflow	The planning loop solves a problem this process does not have
Several model calls per completed task	Re-check latency and cost	Both compound across the loop

Distractor Patterns

Pattern	What it looks like	How to defuse it
Domain 2's list imported	Compliance or constraints offered as the deciding criterion	Nine criteria in 3.1.1; compliance is not among them
A true statement, wrong question	Prompt caching offered where modality decides	Ask which criterion the scenario leaves unsatisfied
Re-testing a satisfied criterion	Cost offered when the scenario said cost is cleared	The answer is never a criterion the question already resolved
Temperature as a correctness dial	"Lower it until the errors stop"	It controls spread; consistency is not correctness
Output ceiling as a style instruction	"Set max tokens low so it summarises"	A ceiling truncates; brevity is a prompt instruction
Length in the wrong sense	The model's maximum confused with the per-request value	Can I change it without changing model?
MCP in a Domain 3 answer	An agent question answered with protocol and patterns	v1.1 moved MCP to Objective 2.1.6
Agent as a better chatbot	"It would improve answer accuracy"	A chatbot answers; an agent acts
Agent for a known sequence	An agent proposed for a fixed monthly process	Known steps means workflow

Scenario Walkthrough

A logistics company wants an assistant that resolves delivery exceptions. Customers write in six languages. Each request sends a 3,500-token operating-procedure document plus the customer's message. Resolving an exception means checking the tracking system, deciding on a remedy, issuing it, and updating the order — and which of those steps is needed varies case by case. The answer must come back inside four seconds. Volume is high and steady.

Requirement	Reading	Decision
Six languages	Multi-lingual — eliminates	Screen candidates on language coverage before anything else
3,500-token fixed prefix, high volume	Prompt caching — ranks	The cost lever for this shape; a smaller model does not address it
Four-second budget, several steps per case	Latency — eliminates, and it compounds	The budget covers the whole loop, not one call
Steps vary case by case	An agent is warranted	The sequence genuinely cannot be enumerated
Issues remedies, updates orders	The agent acts	Design the stop condition and the human review point
High and steady volume	A serving-mode question	Chapter 07's material — not a 3.1.1 criterion

The requirement most people miss is the third. A four-second budget across an agent loop of several model calls is a far harder constraint than four seconds for a single call, and it is the criterion most likely to eliminate the model that won on everything else. The last row is a control: serving mode belongs to Chapter 07, and naming it here is the same domain-drift error as reaching for compliance.

Key Concepts

Term	Definition
Selection criteria (3.1.1)	The nine properties that separate one foundation model from another: cost, modality, latency, multi-lingual, model size, model complexity, customization, input/output length and prompt caching
Modality	The pairing of what a model accepts and what it returns — text, image, audio or video, in each direction independently
Multi-lingual	Whether a model serves every language its users write in; an eliminating criterion, not a score
Model size	How large the model is, and therefore what it costs per token and how fast it responds
Model complexity	How much capability the model brings, against how much the job actually consumes
Customization	Whether and how a model can be adapted to a use case; one criterion here, and the whole subject of Chapter 09
Prompt caching	Reusing the already-processed form of a repeated prompt prefix instead of paying to process it again on every request
Input/output length (as a criterion)	The model's maximum context and response size — a ceiling you shop for and cannot change
Inference parameters (3.1.2)	Request-time settings that change how a model produces its response without changing the model, the prompt, or what it knows
Temperature	The control over how widely the model samples from its distribution of possible next words; it changes spread, not correctness
Maximum output length	A hard stop on generation, which truncates when reached; it is not an instruction to be concise
AI agent	The component that turns a stated goal into a sequence of actions against real systems, planning and acting until the goal is met or a stop condition is reached
Stop condition	The rule that ends an agent's loop when the goal cannot be reached; designed last, and the thing that prevents an agent trying indefinitely

Revision Flashcards

Say the answer aloud before revealing it.

1. Name the nine selection criteria in Objective 3.1.1. → Cost, modality, latency, multi-lingual, model size, model complexity, customization, input/output length, and prompt caching. Four of them eliminate reliably — modality, multi-lingual, input/output length and latency — and the rest mostly rank.

2. Why is quoting Chapter 07's eight factors in a Task 3.1 question a problem? → They are different lists for different decisions. Objective 2.2.3 decides whether generative AI fits and produces a shortlist; Objective 3.1.1 separates models within that shortlist. Compliance and constraints appear on the first list and not the second, so offering compliance as the deciding criterion in a Task 3.1 question is answering the wrong objective.

3. Which three criteria appear on both lists, and why does that matter? → Cost, latency and model complexity. It matters because the overlap is what makes the confusion invisible — a candidate quoting the wrong list is often accidentally right, which is exactly what stops them noticing they have the wrong one.

4. What does it mean to say modality is a pairing rather than a label? → That "supports images" does not say whether images go in, come out, or both. A model that generates images from text and accepts only text is genuinely multi-modal and completely unusable for reading scanned invoices. Name both halves of the pairing you need before screening anything.

5. What shape of workload does prompt caching address, and what is the common wrong instinct? → A large fixed prefix with a short variable part, at volume — the same instruction block or document sent on every request. The wrong instinct is to move to a smaller model instead, which discounts every token rather than removing the repeated processing. The two levers act on different halves of the same bill.

6. "Input/output length" appears in two objectives. What does it mean in each? → In Objective 3.1.1 it is the model's maximum — a ceiling you shop for and cannot change. In Objective 3.1.2 it is the value you set on a given request, inside that ceiling. The test: can I change this without changing model? If not, it is the selection criterion.

7. What does temperature change, and what does it leave untouched? → It changes how widely the model samples from its distribution of possible next words, so it changes how much wording varies between runs. It does not change what the model knows, whether the answer is factually right, or the quality of the source material.

8. A team lowers temperature to zero and the extraction errors continue, now identically every time. What happened? → Exactly what should have been expected. Temperature makes output consistent, not correct. A model that is wrong about something at temperature zero is wrong the same way every time — which is harder to notice than an intermittent error, because it stops looking like a glitch and starts looking like a rule.

9. When is a higher temperature the right choice? → When variety is the deliverable and a human selects afterwards — generating several draft subject lines, ideating options, producing alternatives to choose between. When exactly one output is correct, as in classification or field extraction, variety is a defect.

10. A summary stops mid-sentence. What is the diagnosis, and what is the fix? → The maximum output length was reached; a ceiling truncates rather than instructing. The tell is the stopping point — a model asked to be brief produces something short and complete. If brevity is wanted, ask for it in the prompt.

11. What is the role of an AI agent, in one sentence? → It turns a stated goal into a sequence of actions against real systems, planning a step, taking it, reading the result and deciding the next, until the goal is met or a stop condition is reached. A chatbot answers a question; an agent completes a task.

12. A monthly process has eleven steps, always the same, always in that order. Why is an agent the wrong answer? → Because an agent buys the fact that the sequence need not be known in advance, and this sequence is known. The planning loop adds cost, latency and variability while solving a problem the process does not have. A fixed workflow is cheaper, faster and auditable.

The Five-Beat Answer

The core question this chapter prepares you for: "How would you choose a foundation model for this application, and how would you set it up?"

Five beats, checked in this order. Missing a beat is a failure state — you will be probed on whichever one you skipped.

1. Eliminate first — name the criteria that the scenario leaves unsatisfied, and say that they eliminate rather than rank. Modality as a pairing, language coverage, the maximum lengths against your longest case, and the latency budget.
2. Then rank — cost, model size, model complexity and prompt caching applied to whatever survived. Say explicitly that ranking a candidate that should have been eliminated is the common error.
3. Set the parameters — temperature chosen from whether exactly one output is correct, and the output ceiling set as a safety limit rather than as a request for brevity. State that temperature changes spread and not correctness before you are asked.
4. Decide whether it is an agent at all — an agent if the sequence of steps cannot be known in advance; a workflow if it can. Name what the loop costs: several calls, compounding latency, and a system that acts on real data.
5. Re-check the criteria against the shape you chose — an agentic workload multiplies latency and cost per task, so criteria that were comfortable at one call may bind at twelve.

A strong answer eliminates before it ranks, and names a cost. A weak answer lists criteria.

Why This Helps You

On the job: the expensive mistakes at this stage are quiet ones. A model chosen against the median document rather than the longest, a temperature lowered until failures became reproducible instead of rare, an agent commissioned for a process whose steps were already written down. None of these looks like a mistake in a demonstration, and each is paid for over the life of the system.

In interviews: "how would you pick a model for this?" is a standard screening question, and the answer that separates people is the one that eliminates before it ranks. Saying "modality and language coverage eliminate, then I rank the survivors on cost and caching behaviour" is a structurally different answer from listing nine properties.

On the exam: this is the opening chapter of the largest domain, at 28% of scored content. The highest-value habit is finding the unsatisfied criterion, because Task 3.1 scenarios are constructed so that most criteria are met. The second highest is keeping Domain 2's objectives and Domain 3's apart — three of the questions in this chapter's quiz test only that.

Chapter Checklist

• I can name the nine criteria in Objective 3.1.1 and say which of them eliminate
• I can explain why Objective 2.2.3's eight factors are a different list for a different decision
• I can treat modality as a pairing rather than a label
• I can say what prompt caching addresses, and why a smaller model is not the same lever
• I can distinguish input/output length as a model maximum from input/output length as a request setting
• I can state what temperature changes and what it leaves untouched
• I can explain why a low temperature produces consistency rather than correctness
• I can diagnose a truncated response as an output ceiling rather than a concision instruction
• I can define the role of an agent as completing a task rather than answering about one
• I can name four business applications of agents and what the agent replaces in each
• I can say when a fixed workflow is the better answer than an agent
• I can explain why an agentic workload changes which selection criteria bind

After the Chapter

1. Complete student/project.md — parts 31-33 of the AI/ML Decision Sheet you began in Chapter 01. Bring the same sheet; do not start a new one.
2. Take student/quiz.md closed-book, then review the reasoning for every question you guessed, including the ones you got right.
3. Open the official v1.1 exam guide's Domain 3 page and confirm you can attach a concept from this chapter to objectives 1, 2 and 6 under Task Statement 3.1. While you are there, search the page for "MCP" and "Model Context Protocol" and confirm for yourself that neither appears — then find Objective 2.1.6 on the Domain 2 page, where they live.
4. Next: Chapter 09 — RAG, Vector Stores, and the Customization Cost Ladder (Domain 3, Task 3.1, objectives 3-5). This chapter chose a model and configured it; the model still does not know your organisation's own material. Chapter 09 asks what the cheapest way to fix that is, and turns customization — one criterion among nine here — into the entire decision.

Chapter 09 — RAG, Vector Stores, and the Customization Cost Ladder

Certification Blueprint

Field	Coverage
Exam	AWS Certified AI Practitioner (AIF-C01), exam guide v1.1
Domain	Content Domain 3 — Applications of Foundation Models
Exam weight	28% of scored content — the largest domain on the exam
Task statement	3.1 Design considerations for applications that use foundation models
Objectives	3.1.3 RAG and its business applications · 3.1.4 AWS services that store embeddings in vector databases · 3.1.5 cost tradeoffs of FM customization

What This Chapter Covers

A foundation model was trained on an enormous amount of general text. It has never seen your refund policy, your product catalogue, or the price change that went live on Tuesday. Every question in this chapter follows from that one fact.

There are only two families of answer, and telling them apart is the skill:

	Give it the material	Change the model
What moves	The prompt — new content arrives with each request	The weights — the model itself is different afterwards
When it knows something new	Immediately, as soon as the source changes	After a training cycle completes
What it costs	Tokens per request, plus retrieval infrastructure	Training, hosting, and a permanent maintenance obligation
What it fixes	Missing facts	Wrong behaviour

Almost every wrong answer in Task 3.1 is the second column applied to a first-column problem. It is attractive because it sounds more thorough — and "more thorough" is exactly what the exam punishes here.

What Is Retrieval Augmented Generation?

RAG is fetching relevant material from your own sources at the moment a question is asked, and putting it into the prompt so the model answers from it rather than from memory.

Two ways to hold it:

• An open-book exam. The candidate has not learned more; they have been handed the textbook and told which page. The same person, with the page open, now answers correctly.
• A doctor with your file on the desk. Their training did not change when they picked up your notes. What changed is what is in front of them while they answer.

Both analogies carry the load-bearing point: the person is unchanged. Nothing was learned and nothing was forgotten. That is what makes retrieval immediate and cheap, and it is the property the exam tests hardest.

What to remember from this diagram: every step happens per request, and none of them is a training step. That is why a document published a minute ago is answerable a minute later. Trace the flow once in your own words — question in, vector out, nearest passages back, prompt assembled, answer generated with citations — because a question that garbles any one of those steps is testing whether you can.

What RAG does not do

This definition is examined by its negative space more often than by its positive form.

RAG does not	Because
Change the model's weights	Retrieved text enters the prompt, not the parameters
Teach the model anything permanently	The next request starts from the same unchanged model
Improve the model's writing style or tone	Those are behaviours; retrieval supplies content
Make the model better at reasoning	It supplies material to reason over, not the capability
Remove the need to evaluate	A wrong retrieval produces a confidently wrong grounded answer

A scenario that says "so the model learns our documentation" is describing training, not retrieval. The word learns is very often planted in the distractor.

The last row deserves its own sentence, because it survives into real systems: a citation proves where the text came from, never that it was the right text. If a withdrawn policy is still in the index, an answer citing it looks exactly like a correct one.

RAG in Business Terms

Objective 3.1.3 asks for business applications, not architecture. The pattern repeats: an authoritative internal corpus, changing faster than any training cycle, where a wrong answer is expensive.

Business application	What retrieval provides
Customer support over policies and terms	Current wording, and a citation the agent can show
Internal knowledge search across wikis and tickets	An answer instead of ten links to read
Product and catalogue questions	This week's specification, not last quarter's
Regulated advice, claims and underwriting	Traceability — every statement points at its source
Employee onboarding and HR self-service	One answer that matches the current handbook

Citation is the business feature. In several of these an ungrounded answer is not merely worse, it is unusable — nobody may act on a claim that cannot be traced back to an approved source.

Amazon Bedrock Knowledge Bases

Objective 3.1.3 names this service explicitly. It is managed RAG: the retrieval pipeline offered as a service rather than assembled as an application.

You provide	The service handles
Documents in a data source	Splitting them into passages
A choice of embedding model	Embedding every passage
A vector store, or a managed default	Indexing, and keeping the index current
A question at request time	Retrieval, prompt assembly, and citations

The exam signal is the phrase "without building a retrieval pipeline." A scenario that wants grounded answers over its own documents and does not want to operate the plumbing is naming this service.

What to remember from this diagram: this half runs when documents change, not per request. Confusing it with the retrieval flow above is the most common structural error in this objective — one is ingestion, the other is retrieval, and only the question is embedded per request.

Where Embeddings Live: the Four Named Services

Objective 3.1.4 names four AWS services. They are not four competing products to rank. Each answers a different question about where your data already sits and what shape the query takes.

Service	What it is	The scenario that selects it
Amazon OpenSearch Service	Search and analytics engine with vector search	Search is the primary workload; large corpora; hybrid keyword-plus-vector retrieval
Amazon Aurora	Managed relational database, PostgreSQL-compatible, with vector support	The data is already relational and vectors should sit beside it, at Aurora's scale
Amazon RDS for PostgreSQL	Managed PostgreSQL with vector support	An existing PostgreSQL estate; add vectors without adopting a new data store
Amazon Neptune	Graph database, with vector search alongside graph traversal	Relationships between entities are part of the answer, not just similarity

What to remember from this diagram: the first question is about the data, not the product. A scenario that describes an existing estate has usually already chosen for you. No exam question asks which vector store is best, because none of the four is a bad product — the wrong answers in this objective are true statements about the wrong service for this scenario, which is a much harder distractor to spot than a false one.

Amazon Aurora, and why it is here

Aurora entered the exam's in-scope service list at v1.1. Study material written against the earlier version of the guide will not contain it, and it appears in this objective.

What it is	A managed relational database service, PostgreSQL-compatible, with vector storage and similarity search
Why it appears in an AI objective	Vectors can live in the same database as the rows they describe
What that removes	A second data store to synchronise, secure, back up and pay for
The scenario that names it	Relational data already in Aurora, and a requirement to search it semantically
The pairing to keep straight	Aurora and RDS for PostgreSQL are both PostgreSQL-compatible; the existing footprint decides

Vectors beside the rows they describe is the whole idea. A product's embedding sits in the same database as its price and stock level, and one query reaches both. If you have met Aurora only inside this objective it is easy to assume it must be a purpose-built vector product — it is not, and that assumption breaks the selection signal, which is "the data is already in Aurora."

The Customization Cost Ladder

Customization is any approach that makes a general foundation model behave usefully for your specific situation — from adding a sentence to a prompt, all the way to training a model.

Two ways to hold it:

• Renting a hall for an event. Bring your own decorations; hire a stylist; refit the room; build a venue. Each step costs more, commits you further, and is occasionally the only thing that works.
• Getting a suit. Wear it as sold; have the sleeves taken up; have it re-cut; have one made from scratch. Nobody commissions bespoke tailoring because the sleeves are long.

Objective 3.1.5 does not ask you to perform any of these. It asks you to explain the cost tradeoffs, which is a question about choosing.

Rung	Approach	What actually changes
1	In-context learning	Nothing but the prompt — instructions and examples supplied per request
2	RAG	Still nothing in the model; retrieved source material joins the prompt
3	Fine-tuning	The model's weights, adapted on your examples
4	Distillation	A new, smaller model trained to reproduce a larger one's behaviour
5	Pre-training	A model built from scratch on a large corpus

Rungs 1 and 2 leave the model alone. Rungs 3, 4 and 5 produce a model you now own — and owning a model is a standing obligation to retrain, re-evaluate and host, not a one-off purchase.

What to remember from this diagram: read the labels on the arrows, not the boxes. Each label is a condition that must be true before the climb is justified. If the condition is not established in the scenario, the answer is the rung you are already standing on. Climbing is justified by a condition, not by the failure of the rung below.

What each rung costs, and what it buys

Approach	What it costs	What it buys	The condition that justifies it
In-context learning	Tokens per request; nothing up front	Immediate change, zero commitment	Always try first — it is the baseline
RAG	Retrieval infrastructure, embedding, larger prompts	Current facts and citations, with no training	The model lacks facts, and they change
Fine-tuning	A training job, custom hosting, a maintenance obligation	Behaviour, format and domain style prompting cannot reach	The behaviour is wrong and prompting has genuinely failed
Distillation	Training a student model against a teacher	Lower per-request cost and latency at scale	Volume makes the large model's unit cost the problem
Pre-training	Enormous data, compute and expertise	A model that exists on your terms	No suitable model exists — very rarely the exam's answer

Distillation changes the shape of the cost

The other four rungs simply get more expensive as you climb. Distillation is the one that spends up front to make each request cheaper afterwards, which is why it is the only rung justified by volume rather than by capability.

	Before distillation	After distillation
Model used at request time	Large, capable, expensive per call	Smaller, faster, cheaper per call
Up-front cost	None	A training cycle against the larger model
Where it pays back	—	Only at volume — the saving is per request
What can go wrong	—	The smaller model loses capability the larger one had

Ask what the scenario is complaining about. "The model cannot do X" is not a distillation problem. "The model does X well and we cannot afford it at this volume" is exactly one.

Fresh Facts, or Wrong Behaviour?

What to remember from this diagram: this single question separates most Task 3.1 answers. Facts are retrieved; behaviour is tuned — and if neither is genuinely established by the scenario, the answer is a better prompt. The two failures look very similar in a question stem, which is why the exam can build a pair of near-identical scenarios with opposite answers.

When RAG is not the answer

Retrieval is this task statement's usual answer, which makes the cases where it is wrong worth naming explicitly. A candidate who has learned "always RAG" has simply learned a different wrong rule.

The scenario says	Why retrieval does not fix it	What does
"Responses do not follow our house style"	Style is behaviour; there is no document to retrieve	Prompting first, then fine-tuning
"It must answer in a specialist notation it handles badly"	The capability is missing, not the material	Fine-tuning, or a different model
"Per-request cost is too high at our volume"	Retrieval increases prompt size and cost	Distillation, or a smaller model
"The facts are stable and already well known"	There is nothing to keep current	In-context learning
"Every claim must cite an approved source"	— this is the retrieval case	RAG, and grounding is mandatory

Decision Rules and Exam Signals

Rule 1 — retrieval fills the prompt; training changes the weights. If an option says the model "learns" your documents, it is describing the other family.

Rule 2 — documents are embedded on change, questions on request. Two pipelines, two triggers.

Rule 3 — facts are retrieved, behaviour is tuned. Diagnose before choosing.

Rule 4 — the existing estate picks the vector store. Relationships mean Neptune; already on PostgreSQL means Aurora or RDS for PostgreSQL; search-first at scale means OpenSearch Service.

Rule 5 — the cheapest rung that meets the requirement wins. In-context learning, then RAG, then fine-tuning, then distillation, then pre-training.

Rule 6 — climb on a condition, not on frustration. Each arrow has a stated condition; if the scenario has not established it, do not climb.

Rule 7 — distillation is about volume, not capability. It preserves behaviour at lower unit cost and adds nothing new.

Rule 8 — grounded is not the same as correct. Citation proves provenance; evaluation proves correctness.

Distractor Patterns

Pattern	What it looks like	How to defuse it
"So the model learns our documents"	RAG described as if it trained the model	Retrieval fills the prompt; weights never change
Fine-tuning for fresh facts	Retraining offered for content that changes weekly	Facts are retrieved; a training cycle is slower than the change
Ranking the vector stores	"Which is the best vector database?"	None is; the existing estate and query shape decide
Climbing past the working rung	Fine-tuning offered before prompting was tried	The cheapest approach that meets the requirement wins
Distillation for capability	Offered to fix something the model cannot do	Distillation reduces unit cost; it does not add capability
Pre-training as the thorough answer	Building from scratch for a domain problem	Almost never correct; name what it would actually require
Ingestion confused with retrieval	Embedding described as happening per question	Documents are embedded on change; questions per request
Grounded therefore correct	Citation treated as proof	A stale or wrong passage yields a confidently wrong grounded answer
True statement, wrong service	An accurate description of OpenSearch attached to a graph requirement	All four vector stores are good; match the stated data shape

The fourth and sixth are the two most reliable ways to lose marks in Objective 3.1.5, because the offending option is the one that sounds most rigorous.

Scenario Walkthrough

A hospital group wants an assistant that answers clinician questions about its own treatment protocols. Protocols are revised continuously by a clinical governance committee. Every answer shown to a clinician must be traceable to the approved protocol document it came from. The protocol text is already stored relationally in Aurora alongside approval status and revision metadata. Answers currently read as too informal for clinical use.

Requirement	Reading	Decision
Answers about the group's own protocols	Facts the model has never seen	Retrieval, not training
Protocols revised continuously	Faster than any training cycle	Confirms RAG; rules out fine-tuning for currency
Every answer traceable to its source	Grounding and citation are mandatory	RAG with citations — the business feature, not a nicety
Text already relational in Aurora	The estate has chosen the store	Amazon Aurora as the vector store
Answers read as too informal	Behaviour, not facts	Prompting first; fine-tuning only if that genuinely fails

Five requirements, and the last one is the test. Four of them point at retrieval; the fifth is a different kind of problem sitting inside the same scenario. A candidate who has decided "this is a RAG question" will answer the tone problem with retrieval too — and retrieving the style guide delivers the very instruction that prompting already failed with.

Key Concepts

Term	Definition
Retrieval Augmented Generation (RAG)	Fetching relevant material from your own sources at request time and placing it in the prompt, so the model answers from it rather than from memory; the model's weights are never modified
Grounding	Constraining an answer to retrieved source material, so that every claim traces back to a document rather than to the model's memory
Citation	The reference returned alongside a grounded answer identifying which source passage it came from; proves provenance, not correctness
Ingestion	The pipeline that runs when documents change — splitting into passages, embedding them, and indexing the vectors
Retrieval	The per-request step that embeds the question, finds the nearest stored passages, and returns them for the prompt
Amazon Bedrock Knowledge Bases	Managed RAG — the retrieval pipeline offered as a service, handling splitting, embedding, indexing, retrieval and citation
Amazon OpenSearch Service	A search and analytics engine with vector search; selected when search is the primary workload at large scale
Amazon Aurora	A PostgreSQL-compatible managed relational database that can store vectors beside the relational rows they describe; new to this exam's scope at v1.1
Amazon RDS for PostgreSQL	Managed PostgreSQL with vector support; selected when an existing PostgreSQL estate should gain vectors without a new data store
Amazon Neptune	A graph database offering vector search alongside graph traversal; selected when relationships between entities are part of the answer
In-context learning	Supplying instructions and examples in the prompt itself; rung 1, the cheapest form of customization, with nothing paid up front
Fine-tuning	Adapting a model's weights on your own examples; rung 3, justified when behaviour rather than facts is wrong and prompting has failed
Model distillation	Training a smaller student model to reproduce a larger teacher model's behaviour; rung 4, justified by per-request cost at volume rather than by capability
Pre-training	Building a model from scratch on a large corpus; rung 5, justified only when no existing model fits the domain at all

Revision Flashcards

Say the answer aloud before revealing it.

1. Define RAG in one sentence, and name the thing it leaves unchanged. → Fetching relevant material from your own sources at the moment a question is asked and placing it in the prompt, so the model answers from it rather than from memory. What it leaves unchanged is the model itself — the weights are never modified, which is why a document published a minute ago is answerable a minute later.

2. A colleague says the RAG system means the model "has learned" the company handbook. What is wrong? → Nothing was learned. Retrieved text enters the prompt for that one request and is gone afterwards; the model is byte-identical before and after. There is no threshold of requests at which retrieval turns into training — a system that has served ten million requests has a model in exactly the state it started in.

3. When are source documents embedded, and when are questions embedded? → Documents are embedded during ingestion, which runs when the documents change. Questions are embedded per request, during retrieval. Two pipelines with two different triggers, and confusing them is the most common structural error in this objective.

4. What does a citation prove, and what does it not prove? → It proves provenance — which source passage the answer came from. It does not prove correctness. If a withdrawn policy is still in the index, an answer citing it looks exactly like a correct one, which is why evaluating a RAG application is examined separately.

5. Name the four AWS services in Objective 3.1.4 and the signal that selects each. → Amazon OpenSearch Service when search is the primary workload at large scale; Amazon Aurora when the data is already relational in Aurora and vectors should sit beside it; Amazon RDS for PostgreSQL when there is an existing PostgreSQL estate; Amazon Neptune when relationships between entities are part of the answer rather than just similarity.

6. Why is Amazon Aurora in an AI objective, and what is it actually? → It is a managed PostgreSQL-compatible relational database, not a purpose-built vector product. It appears because vectors can live in the same database as the rows they describe, removing a second data store to synchronise and secure. It entered this exam's scope at v1.1, so older study material will not mention it.

7. List the five customization approaches from cheapest to most expensive. → In-context learning, RAG, fine-tuning, distillation, pre-training. The first two leave the model untouched; the last three produce a model you then own and must retrain, re-evaluate and host.

8. What is the difference between a facts problem and a behaviour problem? → A facts problem is the model not having information — it never saw your policy, your catalogue, your protocol. That is retrieved. A behaviour problem is the model getting format, tone or domain style wrong. That is tuned. They look similar in a question stem and have opposite answers.

9. Why is distillation justified by volume rather than by capability? → Because it spends up front to make each request cheaper afterwards, and that saving only pays back at volume. It compresses behaviour a larger model already has — it adds nothing new. "The model cannot do X" is not a distillation problem; "the model does X well and we cannot afford it at this volume" is.

10. A catalogue changes weekly. Why is weekly fine-tuning the wrong answer? → Because it answers a facts problem with training. Even if every job finished on time, the model would answer from a snapshot averaging half a week old while stock changes hourly, and each cycle buys a permanent maintenance obligation. A scheduling fix cannot repair a category error.

11. Answers are factually right but do not sound like the organisation. Prompting has failed. What now, and why not RAG? → Fine-tune on examples written in the house style, because style is behaviour rather than fact. Retrieving the style guide would deliver a description of the style — the same instruction prompting has already proved insufficient with.

12. When is pre-training the reasonable choice? → When no existing model fits the domain at all. It is justified by the absence of an alternative, not by the difficulty of the problem or by a fine-tuned model missing its quality bar. A question offering it is usually testing whether thoroughness will be mistaken for correctness.

The Five-Beat Answer

The core question this chapter prepares you for: "How would you make a foundation model answer questions about our business, and how far would you go?"

Five beats, checked in this order. Missing a beat is a failure state — you will be probed on whichever one you skipped.

1. Diagnose — say first whether the gap is facts the model never saw or behaviour it gets wrong. Naming this before proposing anything is what separates a designed answer from a remembered one.
2. Retrieve — for a facts problem, describe RAG: fetch from your sources at request time, place it in the prompt, answer from it, cite it. Say explicitly that the model is unchanged, because that is what makes it immediate.
3. Store — name where the embeddings live and why: OpenSearch Service for search-first at scale, Aurora or RDS for PostgreSQL when the data is already relational there, Neptune when relationships decide the answer. Say that the existing estate usually chooses.
4. Ladder — order the five approaches by cost and give the condition on each climb. Name distillation's as volume rather than capability; it is the one most often stated wrongly.
5. Stop — state the cheapest rung that meets the requirement and say what the next one would have cost. Then add that grounding is not evaluation: a citation proves provenance, not correctness.

A strong answer diagnoses before it designs, and names a rung it deliberately did not climb. A weak answer proposes fine-tuning.

Why This Helps You

On the job: the single most expensive avoidable mistake in this area is commissioning a custom model for a problem retrieval would have solved. It looks thorough at the point of approval, and it converts a content problem into a permanent obligation to retrain, re-evaluate and host — while staying, by construction, permanently out of date. The diagnosis in beat one is what prevents it.

In interviews: "when would you fine-tune instead of using RAG?" is a standard screening question, and most candidates answer it as a preference. The strong answer is structural — facts versus behaviour, plus the cost the climb commits you to — and mentioning that a citation proves provenance rather than correctness marks out someone who has actually run one of these systems.

On the exam: Domain 3 is 28% of scored content, the largest on the exam, and these three objectives are the design-decision core of it. The highest-value habit is refusing to climb: when two options both work, the cheaper one is the answer, and the more thorough-sounding option is the trap.

Chapter Checklist

• I can define RAG without describing training, and say what it leaves unchanged
• I can walk the request flow from question to grounded, cited answer
• I can separate ingestion (on document change) from retrieval (per request)
• I can name business applications of RAG and say why citation is the business feature
• I can say what Amazon Bedrock Knowledge Bases removes from a build
• I can name the four vector stores and the scenario signal that selects each
• I can explain why Amazon Aurora appears in this objective, and how it differs from RDS for PostgreSQL
• I can order the five customization approaches by cost and state each climb condition
• I can explain why distillation is justified by volume rather than capability
• I can diagnose whether a scenario describes missing facts or wrong behaviour
• I can say why a grounded answer is not automatically a correct one

After the Chapter

1. Complete student/project.md — parts 34-36 of the AI/ML Decision Sheet you began in Chapter 01. Bring the same sheet; do not start a new one.
2. Take student/quiz.md closed-book, then review the reasoning for every question you guessed, including the ones you got right. Pay particular attention to questions 7 and 9 — they are deliberate mirror images, and missing both means the facts-versus-behaviour diagnosis has not landed.
3. Open the official v1.1 exam guide's Domain 3 page and confirm you can attach a concept from this chapter to the third, fourth and fifth bullets under Task Statement 3.1. Then find Amazon Aurora on the In-Scope AWS Services page and note which category it sits in.
4. Next: Chapter 10 — Prompt Engineering (Domain 3, Task 3.2). This chapter named the bottom rung of the ladder but did not teach it. Chapter 10 is that rung in full: the constructs of a prompt, the techniques that make in-context learning work, and the risks — exposure, poisoning, hijacking and jailbreaking — that arrive as soon as retrieved and user-supplied text share a prompt.

Chapter 10 — Prompt Engineering: Techniques, Practice, and Risk

Certification Blueprint

Field	Coverage
Exam	AWS Certified AI Practitioner (AIF-C01), exam guide v1.1
Domain	Content Domain 3 — Applications of Foundation Models
Exam weight	28% of scored content — the largest domain on the exam
Task statement	3.2 Choose effective prompt engineering techniques
Objectives	3.2.1 concepts and constructs · 3.2.2 techniques · 3.2.3 benefits and best practices · 3.2.4 risks and limitations · 3.2.5 prompt versioning and management with Amazon Bedrock Prompt Management
Service families	Amazon Bedrock Prompt Management

What This Chapter Covers

Chapter 09 ended by placing prompting at the bottom of the customization cost ladder — the rung you try first, and the one that resolves most problems. It did not say how. This chapter is that rung.

Everything in it follows from a single structural fact, and it is worth stating before any terminology:

The model receives one string. Whatever you wrote as instructions, whatever was retrieved from your documents, and whatever the user typed all arrive as a single undifferentiated block of text. There is no channel, no privilege level, no separator the model is obliged to respect.

That one fact explains both halves of this chapter. It is why prompting works at all — you can change what a model does without touching its weights, purely by changing text. And it is why all four of the named risks exist, because anything that reaches the string is read as though the author had written it.

Hold that sentence. The rest of the chapter is its detail.

What to remember from this diagram: the five constructs are things you assemble, but the arrow into the model carries only one payload. The boundaries you can see in the diagram do not exist by the time the model reads it. That is the whole chapter in one picture.

The Five Constructs

Objective 3.2.1 names context, instruction and negative prompts as examples. A complete prompt is usually built from five parts, and being able to name the missing one is more useful on the exam than being able to say a prompt is "too vague".

Construct	What it contributes	What its absence looks like
Context	Background the model would not otherwise have — the role it should adopt, the audience, the situation	Answers that are correct in general but wrong for this setting
Instruction	The task itself, stated as a directive. The one construct a prompt cannot omit	The model summarises when you wanted a comparison
Negative prompt	What to exclude, avoid, or never do	Disclaimers, apologies, or forbidden content that keeps reappearing
Input data	The specific item to act on this time — the ticket, the paragraph, the record	A generic answer that ignores the particular case
Output indicator	The required shape of the response: format, length, schema, structure	Prose where the system needed JSON; a parser that breaks intermittently

The exam signal is diagnostic. A scenario that describes output which is accurate but arrives in the wrong shape is naming a missing output indicator, not a model that needs replacing. A scenario where the model keeps adding a safety disclaimer nobody asked for is naming a missing negative prompt.

Negative prompts deserve their own paragraph

A negative prompt states what must not appear. It is the construct most often left out, and the one most often described incorrectly in distractors.

Two things to hold:

• It is an instruction, not a filter. The model is asked to avoid something; nothing intercepts the output afterwards to check. A distractor that describes a negative prompt as removing content from a response after generation is describing output filtering, which is a Domain 5 control and belongs to Chapter 16.
• It is not a guarantee. Instructing a model to never do something reduces the frequency; it does not make it impossible. This is a limitation of prompt engineering in the sense Objective 3.2.4 asks about, and it is why the risks section later in this chapter matters.

Techniques: How Much Demonstration to Supply

Objective 3.2.2 names five techniques. They are not five alternatives on one list — they sit on two different axes, and the most common exam error is treating them as one.

Axis one — how many examples you supply.

Technique	What you supply	When it is the right choice
Zero-shot	The instruction alone, no examples	The model already does this task reliably. Start here; it is the cheapest per request
Single-shot	One worked example	The task is understood but the shape of a good answer needs pinning down
Few-shot	Several examples, consistently formatted	A pattern needs demonstrating — a classification scheme, a house format, an edge case convention

Axis two — whether you ask for reasoning.

Chain-of-thought asks the model to work through intermediate steps rather than leap to an answer. It is not "more examples"; it is a different request entirely, and it can be combined with any of the three above.

What to remember from this diagram: the branch point is what kind of gap the output has, not how hard the task feels. Every path ends at a template, because a prompt that works is an asset worth reusing rather than retyping.

Why the shot ladder cannot fix a reasoning failure

This is the single highest-value idea in Objective 3.2.2.

Examples demonstrate what a good answer looks like. They do not demonstrate how to get there. If a task fails because it requires several linked deductions — arithmetic, ordering, comparing against multiple constraints — then showing the model more finished answers gives it more targets to imitate and no additional capacity to reason.

The result is a prompt that is longer, more expensive on every single request, and just as wrong.

What to remember from this diagram: the right-hand path produces something the left-hand path never does — a trail. That is a second, quieter benefit of chain-of-thought: when the answer is wrong, you can see which step went wrong. A direct prompt fails silently.

Prompt templates

A template is a prompt with the varying parts parameterised, so the wording that was tested is the wording that ships every time.

A template gives you	Why it matters
One place to change the wording	Otherwise the prompt is copy-pasted across services and drifts
Consistent structure across requests	Output stays parseable because the output indicator is always present
A unit you can version	Which is exactly what Objective 3.2.5 is about

Templates are the bridge to the last objective. Once a prompt is a named, reusable artefact, the question "which version of it is in production?" becomes answerable — and Amazon Bedrock Prompt Management is the service that answers it.

Benefits and Best Practices

Objective 3.2.3 names six things: response quality improvement, experimentation, guardrails, discovery, specificity and concision, and using multiple comments. All six are examinable, and two of them are routinely skipped in study material.

Named item	What it actually means	The practice
Response quality improvement	The headline benefit — better output with no change to the model, no training cost, and effect on the very next request	Treat the prompt as the first lever, not the last
Experimentation	Prompting is cheap enough to try variants and compare them, which is not true of any other rung on the ladder	Change one thing at a time and keep what you tried
Guardrails	In this objective's narrow sense: using the prompt to constrain what the model will do — scope, tone, refusal behaviour	State boundaries in the prompt rather than assuming defaults
Discovery	Using prompting to find out what the model can already do before deciding you need something more expensive	Probe capability first; most "we need fine-tuning" conclusions are untested
Specificity and concision	Say exactly what is wanted, and no more. These pull the same way, not opposite ways	Remove words that do not change the output; add words that do
Using multiple comments	The exam guide's own phrasing — breaking a prompt into clearly delineated commented sections rather than one undifferentiated paragraph	Label the parts, so context, instruction and data are visually separable

⚠️ "Using multiple comments" is the exam guide's literal wording, verified against the live Domain 3 page. It is easy to misread as "multiple components" and it means what it says: annotate and delimit the sections of a prompt. The practical effect is the same thing the constructs table teaches — make the parts distinguishable to a human maintainer, since they are not distinguishable to the model.

Specificity and concision are not in tension, and the exam may test that. A vague prompt and a padded prompt fail for the same reason: the words that would have determined the output are not there, or are buried among words that do nothing. Adding length is not adding specificity.

Risks and Limitations

Objective 3.2.4 names four: exposure, poisoning, hijacking, jailbreaking. Learn them by where the attacker's text enters, because that is the only thing that reliably separates them.

What to remember from this diagram: three arrows go in and one comes out. Three of the risks are about text getting into the prompt; one is about text getting out of it.

Risk	Where it enters	What the attacker achieves
Prompt injection / hijacking	The user input	The model is redirected to the attacker's task instead of yours — "ignore your instructions and do this instead"
Jailbreaking	The user input	The model's safety constraints are escaped, producing content it was built to refuse
Poisoning	The source material the prompt will later retrieve or the examples it is given	The attack is planted in advance and fires when a legitimate request pulls it in
Exposure	Nothing enters — it leaves	The system prompt, private context, or another user's data appears in the response

Telling the confusable pairs apart

Hijacking versus jailbreaking. Both arrive through user input, which is why they are the pair most often confused.

• Hijacking changes the task. The model does something for the attacker rather than for you — it is still behaving, just working for someone else.
• Jailbreaking changes the limits. The model does the thing it was constrained not to do.

A support assistant tricked into writing marketing copy has been hijacked. A support assistant tricked into producing content the operator forbade has been jailbroken.

Poisoning versus hijacking. Both plant instructions, and the difference is timing and route. Hijacking is delivered by the attacker at request time through the input. Poisoning is placed in a document, a knowledge base or an example set beforehand, and is triggered later by an innocent user's ordinary question. That delay is the signal: a scenario where the attacker is not present when the attack fires is describing poisoning.

Exposure is the odd one out and it is the one candidates most often mislabel. Nothing hostile has to enter for exposure to happen — a prompt that includes a system instruction, or context assembled from another customer's record, can simply be repeated back. A scenario in which no attacker appears at all, and confidential text appears in an answer, is exposure.

The limitation, stated plainly

All four exist because of the fact this chapter opened with. The model cannot distinguish your instructions from anything else in the string, so instruction-based defences are advisory. Telling a model to ignore attempts to override it is itself just more text in the same undifferentiated block.

This is why Objective 3.2.4 is a risks and limitations objective rather than a controls objective. The controls — input validation, output filtering, guardrails as a service, least-privilege on what the application can do with the answer — are Domain 5 and are examined in Chapter 16.

Prompt Versioning and Management

Objective 3.2.5 names Amazon Bedrock Prompt Management specifically. The idea underneath it: a prompt is code. It determines system behaviour, it changes, and a change can break production. Everything that follows is what you already do for code.

What to remember from this diagram: the application points at a version, never at the draft. That single arrow is what makes every other property possible.

Capability	What it gives you
Store prompts as named artefacts	The prompt has one home instead of being duplicated in application code
Draft versus version	A draft is editable; a version is an immutable snapshot that cannot change under a running application
Test against saved inputs	A change can be compared before it reaches production
Deploy by version reference	Production behaviour is pinned and reproducible
Roll back	A regression is undone by pointing at the previous version, not by remembering what the wording used to be

The exam signal is "which version was live?" A scenario describing an application whose answers changed and a team that cannot say what the prompt used to say is describing the absence of prompt management. The fix named in this objective is versioning, not more testing and not a better model.

Why deploying a draft is the wrong answer even when it works. A draft is mutable. An application referencing one has behaviour that can change without any deployment, any review, or any record — someone edits the draft and production shifts. The immutability of a version is not a convenience feature; it is the entire point.

Decision Rules and Exam Signals

Rule 1 — the model reads one string. Instructions, retrieved text and user input are indistinguishable by the time they arrive. Every risk in this task statement follows from this.

Rule 2 — name the missing construct. "The prompt is bad" is not a diagnosis. Wrong shape means a missing output indicator; unwanted content means a missing negative prompt; wrong setting means missing context.

Rule 3 — examples fix shape, not reasoning. If the failure is multi-step, few-shot makes the prompt longer and no better. Chain-of-thought is the technique that addresses reasoning.

Rule 4 — start at zero-shot and climb only on evidence. Same discipline as Chapter 09's cost ladder, one level down: every example is paid for on every request.

Rule 5 — chain-of-thought is a different axis from the shot count. It combines with zero-, single- or few-shot; it is not a fourth position on the same ladder.

Rule 6 — specificity and concision agree. Padding is not precision. Remove words that do not change the output.

Rule 7 — classify a risk by where the text entered. User input at request time is hijacking or jailbreaking; planted in advance in a source is poisoning; leaving in the response is exposure.

Rule 8 — task changed is hijacking, limits escaped is jailbreaking.

Rule 9 — prompt defences are advisory, not enforcing. An instruction telling the model to resist override attempts is more text in the same string. Real controls are Domain 5.

Rule 10 — applications reference versions, never drafts. Immutability is what makes rollback and reproducibility possible.

Distractor Patterns

Pattern	What it looks like	How to defuse it
More examples for a reasoning failure	Few-shot offered for a task failing on arithmetic or multi-step logic	Examples show finished answers; they add no reasoning capacity. Chain-of-thought is the technique
Fine-tune it	Training offered before prompting has been genuinely tried	Discovery and experimentation come first; this is Chapter 09's cost ladder restated
Temperature as a prompt technique	Changing an inference parameter offered as prompt engineering	Temperature is Objective 3.1.2 and changes no words in the prompt
Negative prompt as a filter	Described as removing content after generation	It is an instruction before generation. Post-generation removal is output filtering, Domain 5
Hijacking labelled jailbreaking	Any injected-text scenario called jailbreaking	Ask what changed: the task (hijacking) or the limits (jailbreaking)
Poisoning labelled injection	A planted document called prompt injection	Poisoning is placed in advance and fires on an innocent request
Exposure needs an attacker	Assuming a leak implies someone attacked	A prompt can repeat its own system instructions with nobody hostile involved
"Instruct the model to refuse overrides"	Offered as the fix for injection	Advisory only — it is more text in the same undifferentiated string
Longer prompt = more specific	Padding presented as precision	Specificity and concision pull the same way
Deploy the draft	Application pointed at an editable draft	A version is immutable; that is what makes rollback and reproducibility work
Better model instead of versioning	Model swap offered for "answers changed and we don't know why"	That is a change-management gap, and versioning is the named fix

The fifth and sixth rows are the two most reliable ways to lose marks in Objective 3.2.4, because all three risks involve text that was not written by the prompt's author.

Scenario Walkthrough

An insurer runs an assistant that answers policy questions. It retrieves clauses from an internal document library that brokers can upload to. Answers are factually right but arrive as prose, while the downstream system needs structured fields. A separate complaint says the assistant sometimes repeats an internal instruction beginning "You are a policy assistant; never discuss pricing." The team also reports that after someone edited the wording last week, answer quality dropped, and nobody can say what the prompt said before.

Requirement	Reading	Decision
Right facts, wrong shape	The content is fine; the form is unspecified	Missing output indicator — add the required schema
Brokers can upload to the retrieved library	Untrusted text enters the prompt in advance	Poisoning exposure, not hijacking — the uploader need not be present when it fires
Internal instruction appears in answers	The system prompt is leaving in the response	Exposure — and no attacker is implied
Wording edited, quality dropped, no history	Change management, not model quality	Amazon Bedrock Prompt Management — version it, deploy the version, roll back

Four requirements, and the second one is the test. A candidate who has decided "this is a prompt injection question" will label the broker upload as hijacking. It is not: nobody is injecting anything at request time. The attack is stored, and it fires when an ordinary user asks an ordinary question that happens to retrieve it. Route of entry, not intent, is what names these.

Key Concepts

Term	Definition
Prompt engineering	Shaping a model's output by changing the text it is given, without modifying its weights
Context	The construct supplying background the model would not otherwise have — role, audience, situation
Instruction	The construct stating the task as a directive; the one part a prompt cannot omit
Negative prompt	The construct stating what must not appear; an instruction before generation, not a filter after it
Input data	The construct carrying the specific item to act on for this request
Output indicator	The construct specifying the required shape of the response — format, length, schema
Zero-shot	Prompting with an instruction and no examples
Single-shot	Prompting with exactly one worked example, typically to pin down the shape of a good answer
Few-shot	Prompting with several consistently formatted examples to demonstrate a pattern
Chain-of-thought	Asking the model to produce intermediate reasoning steps rather than leap to an answer; addresses reasoning failures, and leaves a trail that can be checked
Prompt template	A prompt with its varying parts parameterised so tested wording is reused rather than retyped
Exposure	A risk in which the system prompt or private context appears in the response; requires no attacker
Poisoning	A risk in which malicious content is planted in advance in a source or example set, and fires when a legitimate request retrieves it
Prompt injection / hijacking	A risk in which text supplied at request time redirects the model to the attacker's task
Jailbreaking	A risk in which crafted input escapes the model's safety constraints, producing content it was built to refuse
Amazon Bedrock Prompt Management	The AWS service for storing, versioning, testing and deploying prompts as managed artefacts
Prompt version	An immutable snapshot of a prompt that an application can reference, enabling reproducible behaviour and rollback

Revision Flashcards

Say the answer aloud before revealing it.

1. State the one structural fact that this whole chapter follows from. → The model receives a single undifferentiated string. Your instructions, any retrieved documents and the user's input are all the same text by the time it reads them — there is no channel, privilege level, or separator it is obliged to respect. That is why prompting can change behaviour without touching weights, and why every risk in this task statement exists.

2. Name the five constructs of a prompt and the one that cannot be omitted. → Context, instruction, negative prompt, input data, output indicator. The instruction is the one a prompt cannot do without — it states the task. The others sharpen, constrain, supply or shape, but without an instruction there is no task.

3. Output is accurate but arrives as prose when the system needed JSON. What is missing? → The output indicator — the construct that specifies the required shape. This is a prompt-construction gap, not a model-quality problem, and the distractor to avoid is the one offering a different or larger model.

4. What is a negative prompt, and what is it not? → It is an instruction stating what must not appear, applied before generation. It is not a filter: nothing inspects the output afterwards to enforce it. An option describing content being removed after generation is describing output filtering, which is a Domain 5 control.

5. Distinguish zero-shot, single-shot and few-shot. → Zero-shot supplies the instruction alone; single-shot adds exactly one worked example, usually to pin down the shape of a good answer; few-shot supplies several consistently formatted examples to demonstrate a pattern. They are one axis — how much demonstration — and every example is paid for on every request.

6. Why can few-shot prompting not fix a multi-step reasoning failure? → Because examples demonstrate what a good answer looks like, not how to reach one. Showing more finished answers gives the model more to imitate and no additional capacity to reason. You get a longer prompt, a bigger per-request bill, and the same wrong answer. Chain-of-thought is the technique that addresses reasoning.

7. What does chain-of-thought produce besides a better answer? → A visible trail. Because intermediate steps appear in the response, a reviewer can see which step went wrong when the answer is wrong. A direct prompt fails silently, which is why chain-of-thought is valuable even when the final answer would have been right.

8. Name the six items in Objective 3.2.3, including the two usually skipped. → Response quality improvement, experimentation, guardrails, discovery, specificity and concision, and using multiple comments. The two usually skipped are discovery — probing what the model can already do before paying for something more expensive — and experimentation, which is only affordable at this rung.

9. Are specificity and concision in tension? → No, they pull the same way. A vague prompt and a padded prompt fail for the same reason: the words that would determine the output are absent or buried. Adding length is not adding specificity. Remove words that do not change the output; add words that do.

10. Distinguish hijacking from jailbreaking. → Both arrive through user input. Hijacking changes the task — the model works for the attacker instead of you. Jailbreaking changes the limits — the model produces something it was constrained to refuse. A support assistant tricked into writing marketing copy was hijacked; one tricked into forbidden content was jailbroken.

11. A broker uploads a document containing hidden instructions; weeks later an ordinary customer question retrieves it and the assistant misbehaves. Name the risk. → Poisoning. The attacker was not present when the attack fired — the malicious text was planted in advance in a source the prompt would later retrieve, and an innocent request triggered it. The delay between planting and firing is the signal that separates it from hijacking.

12. Why must an application reference a prompt version rather than a draft? → Because a version is immutable and a draft is not. An application pointed at a draft has behaviour that can change with no deployment, no review and no record — someone edits the draft and production shifts underneath it. Immutability is what makes reproducible behaviour and rollback possible, and it is the whole point of versioning.

The Five-Beat Answer

The core question this chapter prepares you for: "How would you get better results out of a foundation model without changing the model — and what could go wrong?"

Five beats, checked in this order. Missing a beat is a failure state — you will be probed on whichever one you skipped.

1. Construct — say what a prompt is made of before saying how to improve one: context, instruction, negative prompt, input data, output indicator. Diagnosing a weak prompt means naming the missing part, not calling it vague.
2. Technique — pick from the kind of gap. Zero-shot first; examples for shape and pattern; chain-of-thought for reasoning. Say explicitly that examples cannot repair reasoning, because that is the distinction most candidates miss.
3. Practise — name what good prompting work looks like: experiment cheaply and one change at a time, use discovery to test capability before paying for more, be specific and concise, and delimit the sections of the prompt.
4. Risk — give the four named risks classified by where the text entered: hijacking and jailbreaking through user input, poisoning planted in advance, exposure on the way out. Then state the limitation plainly — the model cannot tell your instructions from anything else, so prompt-level defences are advisory.
5. Manage — finish on versioning. A prompt is code: store it, version it, test it, deploy a version rather than a draft, and roll back when a change regresses. Name Amazon Bedrock Prompt Management.

A strong answer diagnoses the gap before choosing a technique, and names a risk that prompting alone cannot close. A weak answer proposes few-shot for everything.

Why This Helps You

On the job: the most common waste in this area is escalating past prompting without ever testing it properly — commissioning fine-tuning for a problem an output indicator would have solved. The second most common is treating a prompt as a string literal in application code, which is how a team ends up unable to say what changed when quality drops. Both are avoided by the same habit: treat the prompt as an engineered, versioned artefact.

In interviews: "how would you stop prompt injection?" is a standard screening question and most candidates answer it with a better instruction. The strong answer names the structural reason that cannot work — the model reads one undifferentiated string — and moves to controls outside the prompt. Being able to separate hijacking, jailbreaking, poisoning and exposure by route of entry marks out someone who has actually operated one of these systems.

On the exam: Domain 3 is 28% of scored content and this chapter owns an entire task statement within it. The highest-value habits are diagnosing the gap before choosing a technique, and classifying a risk by where the text entered rather than by how the scenario feels.

Chapter Checklist

• I can state why the model receiving one undifferentiated string explains both prompting's power and its risks
• I can name the five constructs and say which one a described failure is missing
• I can explain what a negative prompt is, and why it is not a filter
• I can distinguish zero-shot, single-shot and few-shot, and say what each costs per request
• I can explain why examples cannot fix a multi-step reasoning failure
• I can say what chain-of-thought provides beyond a better answer
• I can explain what a prompt template gives me and how it leads to versioning
• I can name all six items in the benefits and best practices objective
• I can explain why specificity and concision are not in tension
• I can classify exposure, poisoning, hijacking and jailbreaking by where the text entered
• I can tell hijacking from jailbreaking by whether the task or the limits changed
• I can explain why prompt-level defences against injection are advisory
• I can say why an application must reference a version rather than a draft

After the Chapter

1. Complete student/project.md — parts 37-39 of the AI/ML Decision Sheet you began in Chapter 01. Bring the same sheet; do not start a new one.
2. Take student/quiz.md closed-book, then review the reasoning for every question you guessed, including the ones you got right. Pay particular attention to questions 8 and 10 — they are deliberate mirror images, and missing both means the route-of-entry classification has not landed.
3. Open the official v1.1 exam guide's Domain 3 page and confirm you can attach a concept from this chapter to each of the five bullets under Task Statement 3.2. Note the exact wording of the third bullet — it says "using multiple comments", and reading it as "components" changes what it asks for.
4. Next: Chapter 11 — Training and Fine-Tuning Foundation Models (Domain 3, Task 3.3). This chapter's boundary is the point where prompting has genuinely failed. Chapter 11 starts there: what pre-training, fine-tuning, continuous pre-training and distillation actually involve, and the data preparation each one demands.

Chapter 11 — Training and Fine-Tuning Foundation Models

Certification Blueprint

Field	Coverage
Exam	AWS Certified AI Practitioner (AIF-C01), exam guide v1.1
Domain	Content Domain 3 — Applications of Foundation Models
Exam weight	28% of scored content — the largest domain on the exam
Task statement	3.3 Describe the training and fine-tuning process for FMs
Objectives	3.3.1 key elements of training an FM · 3.3.2 methods for fine-tuning an FM · 3.3.3 how to prepare data to fine-tune an FM

What This Chapter Covers

Chapter 09 gave you a ladder. In-context learning, RAG, fine-tuning, distillation, pre-training — ordered by cost, each with a condition that must be true before you climb. That chapter deliberately refused to explain the top rungs. It only ranked them.

This chapter opens them.

There is one question that separates every method in this task statement, and it is not about expense or sophistication:

What data does it consume, and what does it change?

Answer those two things and every method here becomes distinguishable. Miss them and the four names blur into "training, but more so" — which is exactly the confusion the exam's distractors are built from.

Method	What data it consumes	What it changes
Pre-training	An enormous general corpus, unlabelled	Creates the weights from nothing
Continuous pre-training	Your raw domain text, unlabelled	Extends existing weights with new subject matter
Fine-tuning	Your example pairs, labelled	Adjusts existing weights toward a task or behaviour
Distillation	The larger model's own outputs	Produces a different, smaller model

The labelled/unlabelled split in the middle column is the highest-value distinction in this task statement. It is what separates continuous pre-training from fine-tuning, and those two are the pair the exam most often asks you to tell apart.

The Four Key Elements of Training

Objective 3.3.1 names four operations. They are not four intensities of the same thing — they are four different jobs, and only two of them are usually available to an organisation.

What to remember from this diagram: follow the dashed arrows, not the solid ones. The solid path is the full sequence; the dashed ones are the skips, and almost every real project takes them. A scenario that describes an organisation walking the whole solid path from the left is describing something very few organisations ever do.

Pre-training

Pre-training is building a foundation model from scratch — starting with no weights at all and learning language from an enormous general corpus.

What it consumes	Vast quantities of general text, unlabelled — the model learns by predicting what comes next
What it produces	A model that did not previously exist
What it costs	Enormous data, compute and specialist expertise
Who does it	Model providers, overwhelmingly — not their customers
The scenario that names it	No existing model fits the domain at all

On the exam, pre-training is almost never the correct answer, and it is offered often. It is the option that sounds most rigorous, which is precisely what makes it a reliable trap. When you see it, ask what it would actually require — and whether the scenario has established that no existing model could work.

Continuous pre-training

Continuous pre-training is taking an existing foundation model and continuing to train it on your own raw text, so it absorbs the vocabulary and conventions of your domain.

What it consumes	Your raw, unlabelled domain text — documents, transcripts, filings, manuals
What it produces	The same model, extended with domain familiarity
What it does not need	Labels. There are no correct answers to supply
The scenario that names it	The model does not speak the language of the field

This is the method most often missed, because it sits between the two everyone knows. It uses pre-training's mechanism (raw text, no labels) but starts from an existing model like fine-tuning does.

⚠️ A genuine quirk of the v1.1 exam guide worth knowing: continuous pre-training is listed twice — as a key element of training in Objective 3.3.1, and as a method for fine-tuning an FM in Objective 3.3.2. That is not an error in your reading. It is a real boundary case, and the guide places it on both sides. If a question treats it as a fine-tuning method, that is consistent with the guide; if another treats it as a distinct training element, so is that.

Fine-tuning

Fine-tuning is adapting an existing model's weights using your own labelled examples, so it performs a specific task or behaves in a specific way.

What it consumes	Labelled example pairs — an input and the response you wanted
What it produces	The same model, adjusted toward your task
What it demands	Someone to produce the labels, which is the real cost
The scenario that names it	Behaviour, format or task performance is wrong, and prompting has genuinely failed

Distillation

Distillation is training a smaller student model to reproduce a larger teacher model's behaviour, so that requests can be served more cheaply.

What it consumes	The teacher model's own outputs — generated, not hand-labelled
What it produces	A different, smaller model — this is the only method here that does
What it buys	Lower cost and latency per request
What it risks	The student loses capability the teacher had
The scenario that names it	The model works well and cannot be afforded at the required volume

Distillation is the only one of the four justified by volume rather than by capability. Chapter 09 made this point about cost; it is worth repeating as a mechanism. Distillation compresses behaviour that already exists. It cannot add any.

Fine-Tuning or Continuous Pre-Training?

This is the pair the exam most reliably asks you to separate, and one question does it.

What to remember from this diagram: the split is language versus task. Continuous pre-training teaches the model what the words mean in your field; fine-tuning teaches it what to do when asked. And the practical tell follows from that: if the scenario describes a pile of documents, that is continuous pre-training's input. If it describes examples of good answers, that is fine-tuning's.

	Continuous pre-training	Fine-tuning
Data required	Raw domain text	Input-and-response pairs
Labels needed	No	Yes
What it teaches	The domain's language	The required task or behaviour
Typical source	Existing corpus you already hold	Must usually be created
The complaint it answers	"It does not understand our terminology"	"It understands, but answers wrongly"

The labelling requirement is the practical difference, not the algorithm. An organisation with ten years of unlabelled filings can begin continuous pre-training immediately. The same organisation may need months to produce fine-tuning pairs — because someone has to write the correct answers.

Methods for Fine-Tuning

Objective 3.3.2 names four methods. Three are genuinely distinct approaches; the fourth is continuous pre-training, appearing here for the second time.

What to remember from this diagram: the last box is the one that matters most. "Re-check the diagnosis" is a real exam answer far more often than any of the three methods above it — because most scenarios that look like tuning problems are prompting problems that were never properly tried.

Method	What it is	The scenario that selects it
Instruction tuning	Training on pairs of instruction and desired response, so the model reliably follows directions and answers in the required form	Answers are correct but ignore the instruction's form, tone or structure
Domain adaptation	Adapting a general model to a specialist field's terminology, conventions and expectations	The field has its own vocabulary the general model handles badly
Transfer learning	Reusing what a model learned on one task as the starting point for a related task	A related, already-learned capability exists to build on
Continuous pre-training	Continuing to train on raw domain text without labels	A large body of domain text exists and the gap is language rather than task

Transfer learning is the principle, not a separate product

This is worth stating plainly because the exam's phrasing can make it sound like a fourth tool sitting beside the others.

Transfer learning is the general idea that knowledge learned for one task can be reused for another. Fine-tuning a foundation model is an application of transfer learning — the model transferred what it learned in pre-training to your task. When the guide lists it as a method, it is naming the principle. A question describing "reusing a model trained on one task as the starting point for a related one" is describing transfer learning by definition.

Instruction tuning versus domain adaptation

The scenario says	Reading	Method
"It gives good information but ignores the format we asked for"	Instruction-following failure	Instruction tuning
"It does not know what our industry's terms mean"	Vocabulary gap	Domain adaptation or continuous pre-training
"It answers legal questions like a general assistant, not a lawyer"	Domain conventions	Domain adaptation
"We have a model for a near-identical task already"	Reuse an existing capability	Transfer learning
"The tone is wrong and we have never written a prompt for it"	Nothing has been tried	Neither — prompt first

Preparing the Data

Objective 3.3.3 is the objective candidates under-prepare, and it carries as many marks as the other two. It names six things: data curation, governance, size, labeling, representativeness, and RLHF.

The training is not the project. The data preparation is the project. A fine-tuning job is hours of compute against months of assembling, cleaning, labelling and checking what goes into it. A scenario that treats the training run as the hard part has usually mislocated the difficulty.

What to remember from this diagram: the dashed arrow going backwards is the honest part. Representativeness is checked near the end and routinely sends you back to curation — which is why data preparation is iterative and why estimating it as a single up-front step is how these projects overrun.

Step	What it means	The failure it prevents
Curation	Selecting and cleaning what goes in; removing duplicates, errors and irrelevant material	Training on noise, and teaching the model mistakes that were in the source
Governance	Consent, licensing, privacy, and permission to use the data for this purpose	Training on data you had no right to use — which cannot be undone once it is in the weights
Size	Enough examples for the task; more is not automatically better if quality falls	A model that has not seen enough of the task to generalise
Labeling	Marking the correct response for each example	Fine-tuning without a target; this is the step that makes fine-tuning expensive
Representativeness	Does the data reflect the real range of users, cases and conditions?	A model that works for the majority case and fails everyone outside it
RLHF	Using ranked human judgement to align the model with what people actually prefer	A model that is technically correct and unhelpful

Governance cannot be fixed afterwards

Of the six, governance is the one with no remedy. Retrieved content can be removed from an index in minutes. Training data cannot be removed from weights. Once a model has been trained on material you had no licence, consent or lawful basis to use, the model itself is the problem, and the remedy is retraining — which means the cost of the mistake is the cost of the whole job again.

This is why governance appears in a technical objective about data preparation rather than only in Domain 5. It is a precondition, not a review step.

Representativeness is the one that causes harm

Size and representativeness are easily confused, and the exam separates them.

• Size asks how much.
• Representativeness asks of what.

A large dataset drawn entirely from one region, one customer segment or one time period is big and unrepresentative at the same time. The resulting model performs well on average and fails specific groups — which is where fine-tuning turns into a fairness problem. Chapter 14 examines that consequence under responsible AI; this objective examines the preparation step that prevents it.

Reinforcement learning from human feedback

RLHF is using human judgement about which model outputs are better to train the model toward what people actually prefer.

What to remember from this diagram: humans rank outputs; they do not write them. That is the detail most often stated wrongly. RLHF does not require someone to author the perfect answer — it requires someone to say which of several candidates is better, which is a far cheaper judgement to make and a far harder one to scale.

What humans supply	Preference — a ranking of candidate outputs
What humans do not supply	The ideal answer text itself
What the reward model does	Learns to predict human preference, so it can score outputs at scale
What it is for	Alignment with what people find helpful — not factual accuracy
Why it appears in Objective 3.3.3	It is a data preparation approach: the preferences are the data

RLHF fixes helpfulness, not correctness. A model aligned to human preference produces answers people like. Whether those answers are true is a different question, examined in Chapter 12.

Decision Rules and Exam Signals

Rule 1 — ask what data it eats and what it changes. Those two properties identify every method in this task statement.

Rule 2 — labels are the dividing line. Raw text means continuous pre-training. Input-and-response pairs mean fine-tuning.

Rule 3 — distillation makes a new, smaller model; nothing else here does. The other three modify or create weights for the same model.

Rule 4 — pre-training is almost never the answer. It is justified by the absence of any suitable model, never by the difficulty of the problem.

Rule 5 — "re-check the diagnosis" beats every tuning method when the scenario has not established that prompting failed.

Rule 6 — the data preparation is the project. If a scenario asks what will take the longest, it is the labelling and curation, not the training run.

Rule 7 — governance has no undo. Data in weights cannot be withdrawn; only retraining removes it.

Rule 8 — size and representativeness are different questions. How much, versus of what.

Rule 9 — in RLHF humans rank, they do not write. Preference data, not model answers.

Rule 10 — RLHF aligns helpfulness, not truth. Accuracy is Chapter 12's subject.

Distractor Patterns

Pattern	What it looks like	How to defuse it
Fine-tuning offered with unlabelled data	"Fine-tune the model on the company's document archive"	Fine-tuning needs labelled pairs; a raw archive is continuous pre-training's input
Continuous pre-training described as needing labels	"Continue pre-training on labelled question-and-answer pairs"	The labels are the giveaway — that describes fine-tuning
Pre-training as the thorough answer	Building from scratch for a domain problem	Almost never correct; ask what it would actually require
Distillation to add capability	Offered to make the model do something new	Distillation compresses existing behaviour; it adds nothing
Tuning before prompting	A training method offered with no evidence prompting was tried	The cheapest approach that meets the requirement wins
RLHF as answer-writing	"Experts write ideal responses for the model to learn from"	That is supervised fine-tuning; RLHF collects rankings
RLHF for factual accuracy	Offered to stop the model being wrong	It aligns preference, not truth
Size offered as the fix for representativeness	"Collect more data" for a model failing one user group	More of the same skew is still skewed
Governance treated as a later review	Compliance sign-off scheduled after training	Training on unlicensed data cannot be undone without retraining
Transfer learning as a distinct product	Presented as a fourth tool beside the others	It is the underlying principle; fine-tuning is an instance of it

The first two are a matched pair and the most reliable marks in this task statement. Both are answered by looking at one thing: is the data labelled?

Scenario Walkthrough

A national insurer wants a claims assistant. Its models handle general English well but consistently misread the insurer's policy terminology, in which ordinary words carry specific contractual meanings. The insurer holds twenty years of claims correspondence and adjuster reports, none of it annotated. It also wants answers structured to a fixed internal template; prompting has been tried at length and the structure is still inconsistent. Legal has not yet confirmed whether historical correspondence may be used for model training. Most historical claims come from two regions where the insurer has operated longest.

Requirement	Reading	Decision
Misreads domain terminology	A language gap, not a task gap	Continuous pre-training
Twenty years of unannotated text	Raw, unlabelled — exactly its input	Confirms continuous pre-training; rules out fine-tuning for this part
Fixed template, prompting genuinely failed	A behaviour gap, and the precondition is met	Instruction tuning, needing labelled pairs
Legal has not confirmed permission	Governance, and it is a blocker	Resolve before training — it cannot be undone afterwards
Claims concentrated in two regions	Representativeness, not size	Check coverage before training, or the model fails other regions

Five requirements, two different training methods, and one that stops the project. The governance row is the test. A candidate optimising the technical answer will sequence the two training methods correctly and start immediately — and training on correspondence the insurer may not lawfully use puts the unusable data somewhere it cannot be removed from.

Note also what the last row is not: the fix for a two-region skew is not "collect more claims." It is collecting claims from the other regions.

Key Concepts

Term	Definition
Pre-training	Building a foundation model from scratch on an enormous unlabelled general corpus; produces a model that did not previously exist
Continuous pre-training	Continuing to train an existing model on raw, unlabelled domain text so it absorbs a field's vocabulary and conventions; listed in the guide under both Objective 3.3.1 and Objective 3.3.2
Fine-tuning	Adapting an existing model's weights using labelled input-and-response pairs so it performs a specific task or behaves in a specific way
Distillation	Training a smaller student model to reproduce a larger teacher model's behaviour; the only method here that produces a different model, and justified by per-request cost at volume
Instruction tuning	Fine-tuning on pairs of instruction and desired response so the model reliably follows directions and answers in the required form
Domain adaptation	Adapting a general model to a specialist field's terminology, conventions and expectations
Transfer learning	The principle that knowledge learned for one task can be reused as the starting point for a related task; fine-tuning a foundation model is an instance of it
Data curation	Selecting and cleaning what enters a training set — removing duplicates, errors and irrelevant material
Data governance	Establishing consent, licensing, privacy and lawful basis for using data in training; the one preparation step with no remedy after the fact
Representativeness	Whether training data reflects the real range of users, cases and conditions; distinct from size, and the property whose absence produces a model that fails specific groups
Labeling	Marking the correct response for each training example; the step that makes fine-tuning expensive, because it requires human effort per example
Reinforcement learning from human feedback (RLHF)	Training a model toward human preference by having people rank candidate outputs, learning a reward model from those rankings, and updating the model to score well against it
Reward model	A model trained on human preference rankings that can then score outputs at scale, standing in for the human judgement it learned from
Catastrophic forgetting	The loss of previously held general capability when a model is trained heavily on narrow new data

Revision Flashcards

Say the answer aloud before revealing it.

1. Name the two questions that identify every training method in this task statement. → What data does it consume, and what does it change? Pre-training eats an enormous unlabelled general corpus and creates weights from nothing; continuous pre-training eats your raw unlabelled domain text and extends existing weights; fine-tuning eats labelled example pairs and adjusts existing weights; distillation eats the teacher model's own outputs and produces a different, smaller model.

2. What single property separates continuous pre-training from fine-tuning? → Labels. Continuous pre-training takes raw, unlabelled domain text and teaches the model the field's language. Fine-tuning takes labelled input-and-response pairs and teaches the model a task or behaviour. If a scenario describes an archive of documents, that is continuous pre-training's input; if it describes examples of good answers, that is fine-tuning's.

3. Why is pre-training almost never the right exam answer, and when is it right? → Because it is justified only by the absence of any suitable existing model, never by the difficulty of the problem. It requires enormous data, compute and specialist expertise, and is done overwhelmingly by model providers rather than their customers. It is offered often precisely because it sounds the most thorough.

4. What does distillation do that none of the other three methods do? → It produces a different, smaller model. The other three either create or modify the weights of the model in question; distillation trains a new student model to reproduce a teacher's behaviour, and it is justified by per-request cost at volume rather than by any capability it adds. It cannot add capability at all.

5. Continuous pre-training appears in two different objectives. Is that a mistake? → No. The v1.1 guide lists it as a key element of training in Objective 3.3.1 and again as a method for fine-tuning an FM in Objective 3.3.2. It is a genuine boundary case: it uses pre-training's mechanism of raw unlabelled text but starts from an existing model the way fine-tuning does. Both framings are consistent with the guide.

6. Distinguish instruction tuning from domain adaptation. → Instruction tuning trains on instruction-and-response pairs so the model follows directions and answers in the required form — it fixes a model that gives good information in the wrong shape. Domain adaptation adapts the model to a specialist field's terminology and conventions — it fixes a model that does not know what the field's terms mean.

7. What is transfer learning, and why does it feel like an odd item on the list? → It is the general principle that knowledge learned for one task can be reused as the starting point for a related one. It feels odd beside the others because it is not a separate product — fine-tuning a foundation model is itself an application of transfer learning, since the model transfers what it learned during pre-training to your task.

8. Name the six things Objective 3.3.3 lists for preparing fine-tuning data. → Data curation, governance, size, labeling, representativeness, and reinforcement learning from human feedback. The objective carries as many marks as the other two in this task statement and is the one candidates under-prepare because it sounds administrative.

9. Why is governance the preparation step with no remedy? → Because training data cannot be removed from weights. Content in a retrieval index can be deleted in minutes; material a model was trained on is embedded in the model itself, and the only way to remove it is to retrain — meaning the cost of the mistake is the cost of the entire job again. That is why it is a precondition rather than a review step.

10. What is the difference between size and representativeness? → Size asks how much data; representativeness asks what it is data of. A dataset drawn entirely from one region or customer segment can be very large and completely unrepresentative. The resulting model performs well on average and fails everyone outside the majority case, and the fix is not more of the same data but data from the missing groups.

11. In RLHF, what exactly do the humans provide? → Rankings. They compare candidate outputs the model produced and say which is better. They do not write ideal answers — that would be supervised fine-tuning. Those rankings train a reward model, which can then score outputs at scale, and the model is updated to score well against it.

12. What does RLHF align, and what does it not? → It aligns the model with what humans find helpful and prefer. It does not make the model factually correct. A model tuned to preference produces answers people like; whether they are true is a separate question, and evaluating that is Chapter 12's subject.

The Five-Beat Answer

The core question this chapter prepares you for: "We want to train a model on our own data. How would you approach it?"

Five beats, checked in this order. Missing a beat is a failure state — you will be probed on whichever one you skipped.

1. Diagnose — say what is actually wrong before naming a method. Is the gap the domain's language, the required task or behaviour, or per-request cost? Each points at a different method, and stating this first is what separates a designed answer from a remembered one.
2. Check the precondition — say whether prompting and retrieval have genuinely been tried. If they have not, that is the answer, and any training method is premature.
3. Match the method to the data you have — raw unlabelled text supports continuous pre-training today; fine-tuning needs labelled pairs that usually do not exist yet. Say which you hold, because it decides what is actually available rather than merely appropriate.
4. Name the data work — curation, governance, size, labelling and representativeness, and say that this is the majority of the effort. Put governance first and say why: it cannot be fixed after training.
5. Say how you would know it worked — and hand off to evaluation. Naming that you would evaluate, rather than assuming a completed training run is a success, is the beat that most answers omit.

A strong answer diagnoses before it selects, and treats the data as the project. A weak answer names a training method in the first sentence.

Why This Helps You

On the job: the most common way these projects fail is not a bad training run. It is discovering at the labelling stage that the data required does not exist, or discovering after training that the organisation never had permission to use it. Both are visible in advance, and both are found by the diagnosis in beats one and four rather than by any technical step.

In interviews: "when would you fine-tune versus continue pre-training?" separates candidates who have run one of these from those who have read about them. The strong answer is about the data — labelled pairs versus raw text — and about which one the organisation actually has. Mentioning that humans rank rather than write in RLHF is another reliable marker.

On the exam: Domain 3 is 28% of scored content, and this task statement's three objectives split their marks fairly evenly — which means the data preparation objective is worth as much as the two about methods. Candidates who study the method names and skip the preparation step lose a third of the available marks to an objective they considered administrative.

Chapter Checklist

• I can name the four key elements of training and say what data each consumes
• I can say what each of the four changes, and which one produces a different model
• I can separate continuous pre-training from fine-tuning by the labelling requirement
• I can explain why continuous pre-training appears in two objectives
• I can say when pre-training is genuinely the right choice, and why it usually is not
• I can explain why distillation cannot add capability
• I can distinguish instruction tuning, domain adaptation and transfer learning
• I can name all six data preparation elements from Objective 3.3.3
• I can explain why governance cannot be remedied after training
• I can distinguish size from representativeness and say what each failure looks like
• I can describe the RLHF loop and say what humans actually supply
• I can say what RLHF aligns and what it does not

After the Chapter

1. Complete student/project.md — parts 40-42 of the AI/ML Decision Sheet you began in Chapter 01. Bring the same sheet; do not start a new one.
2. Take student/quiz.md closed-book, then review the reasoning for every question you guessed, including the ones you got right. Pay particular attention to questions 3 and 4 — they are deliberate mirror images, and missing both means the labelled-versus-unlabelled split has not landed.
3. Open the official v1.1 exam guide's Domain 3 page and confirm you can attach a concept from this chapter to each of the three bullets under Task Statement 3.3. Note where continuous pre-training appears, and satisfy yourself that it is listed under both 3.3.1 and 3.3.2.
4. Next: Chapter 12 — Evaluating Foundation Models (Domain 3, Task 3.4). This chapter produced a model; it gave you no way to know whether it is any good. Chapter 12 is that: human-in-the-loop evaluation, benchmark datasets, Amazon Bedrock Model Evaluation, the metric families including ROUGE, BLEU and BERTScore, and the business alignment metrics that decide whether the model earned its place at all.

Chapter 12 — Evaluating Foundation Models, and Proving Business Value

Certification Blueprint

Field	Coverage
Exam	AWS Certified AI Practitioner (AIF-C01), exam guide v1.1
Domain	Content Domain 3 — Applications of Foundation Models
Exam weight	28% of scored content — the largest domain on the exam
Task statement	3.4 Describe methods to evaluate FM performance
Objectives	3.4.1 approaches to evaluate · 3.4.2 metrics · 3.4.3 meeting business objectives · 3.4.4 evaluating FM applications · 3.4.5 business objective alignment metrics
Service families	Amazon Bedrock Model Evaluation

What This Chapter Covers

Chapter 11 ended with a model. Pre-trained, fine-tuned, distilled — whichever route was taken, the output was an artefact, and the chapter deliberately gave you no way at all to know whether it was any good.

This chapter is that answer, and it completes Domain 3.

Everything in it follows from a single structural fact, and it is worth stating before any metric name appears:

Every metric in this chapter scores a proxy, never the thing you actually care about. ROUGE counts overlap with a reference summary — not usefulness. A benchmark scores a curated dataset — not your traffic. Even a human reviewer scores the sample they were shown. The skill this task statement tests is knowing which proxy you are holding and what it structurally cannot see.

That one fact explains the whole chapter. It is why several different metrics exist rather than one good one. It is why an FM application needs its own evaluation separate from the model inside it. And it is why Objectives 3.4.3 and 3.4.5 exist at all — because a number that measures a proxy can look excellent while the deployment it describes has changed nothing.

Hold that sentence. The rest of the chapter is its detail.

What to remember from this diagram: there are three layers, not one, and each is blind to the one below it in importance. A model score cannot see your users. An application score cannot see whether the task was worth doing. Only the bottom layer answers the question that authorised the budget. Most exam questions in this task statement are really asking which layer the scenario is about.

Approaches to Evaluating an FM

Objective 3.4.1 names three approaches: human-in-the-loop evaluation, benchmark datasets, and Amazon Bedrock Model Evaluation. They are not three competing products — the first two are methods, and the third is the AWS service that runs both.

Human-in-the-loop evaluation

People read model outputs and judge them.

Property	Detail
What it gives you	The only genuine ground truth for subjective quality — tone, helpfulness, safety, nuance, whether an answer is actually useful
What it costs	Slow and expensive, and it does not scale with traffic
What it is blind to	Whatever was not in the sample. Reviewers see a slice, and a rare failure mode can sit entirely outside it

Use it when the quality question is a judgement, not a comparison against a known correct answer. "Is this refusal appropriately worded?" has no reference string to match against.

Benchmark datasets

A standardised set of tasks with known good answers, run against a model to produce a comparable score.

Property	Detail
What it gives you	Repeatability and comparability. Two models measured the same way can be ranked
What it costs	Little, once it exists — this is the cheap, automatable end
What it is blind to	Your workload. A benchmark is somebody else's data. Strong benchmark performance is evidence about the benchmark

⚠️ The most examinable limitation on this page. A benchmark tells you how a model does on the benchmark. It does not tell you how it will do on your documents, your customers' phrasing, or your edge cases. A scenario in which a model "scores well on standard benchmarks but underperforms in production" is not describing a contradiction — it is describing exactly what benchmarks are for and what they are not.

Amazon Bedrock Model Evaluation

The named AWS service. It matters that it runs both kinds of evaluation:

Mode	What happens
Automatic evaluation	Bedrock runs the model against datasets — built-in or your own — and computes metric scores without a human in the path
Human evaluation	Bedrock manages a workflow in which people score outputs, using either your own work team or an AWS-managed one

The point for the exam is that choosing Bedrock Model Evaluation does not choose a method. It is where both methods are run and compared. A distractor that treats it as purely automatic — or purely human — has misread the service.

What to remember from this diagram: the branch point is does a reference answer exist. That single question separates the automatable half of this objective from the half that needs judgement, and it is the question the exam asks in scenario form.

The Metrics

Objective 3.4.2 names four: ROUGE, BLEU, BERTScore, and LLM-as-a-judge. The exam expands the first two, and the expansions are worth knowing because they encode what each one measures:

• ROUGE — Recall-Oriented Understudy for Gisting Evaluation
• BLEU — Bilingual Evaluation Understudy

Read those two names slowly. "Recall-oriented" and "Bilingual" tell you almost everything.

ROUGE

Measures overlap between the generated text and a reference, oriented toward recall: of the material in the reference, how much did the candidate capture?

Built for summarization. A good summary is one that kept the important content, so recall — did you keep it? — is the natural orientation.

BLEU

Measures overlap too, but oriented toward precision: of the material in the candidate, how much of it appears in the reference?

Built for translation. The clue is in the name — Bilingual Evaluation Understudy. A good translation should not introduce content the source did not contain, so precision — is what you produced supported? — is the natural orientation.

ROUGE and BLEU are the confusable pair in this objective, and the exam knows it. Both count n-gram overlap against a reference. The separation to memorise: ROUGE is recall-oriented and its home task is summarization; BLEU is precision-oriented and its home task is translation. If you can only hold one thing, hold the task: summaries → ROUGE, translation → BLEU.

BERTScore

Instead of counting matching words, BERTScore compares embeddings — it scores semantic similarity.

This matters because ROUGE and BLEU share a structural weakness: a paraphrase that is completely correct but uses different words scores badly. "The flight was cancelled" and "They called off the flight" have almost no n-gram overlap and identical meaning. BERTScore is the named metric that addresses exactly this.

Exam signal: a scenario that stresses "correct but worded differently", "paraphrase", or "semantically equivalent" is pointing at BERTScore.

LLM-as-a-judge

A foundation model scores another model's output against stated criteria.

Property	Detail
What it gives you	Human-style judgement at machine scale, on open-ended tasks where no reference answer exists
What it costs	Inference cost per judgement, and design effort on the criteria
What it is blind to	Its own limitations. The judge is a foundation model, with the same capacity for bias, inconsistency and error as the model under test

This is the v1.1-era metric most likely to appear in a scenario about open-ended output — assistants, creative generation, multi-turn conversation — where there is nothing to diff against.

What to remember from this diagram: you route by task shape, not by which metric sounds most sophisticated. And every path ends in the same warning — the one this chapter opened with.

What all four are blind to

None of these metrics knows whether the answer was useful, true, or worth producing.

• A summary can score highly on ROUGE and omit the one clause that mattered legally.
• A translation can score highly on BLEU and be unusable in register.
• BERTScore rewards semantic similarity to a reference — including similarity to a reference that was itself wrong.
• An LLM judge can confidently prefer a fluent, incorrect answer over an awkward, correct one.

Veracity is the hardest gap. A confident hallucination is fluent, well-formed, and often close in embedding space to the truth. That is why hallucination detection is treated as its own discipline in Domain 5 (Chapter 16) rather than as a metric here.

Evaluating Applications, Not Just Models

Objective 3.4.4 is the objective most often skipped in study material, and it carries a distinct idea: an FM application is not an FM.

By the time a user sees an answer, the model was one component among several. A RAG system retrieved first. An agent chose tools and sequenced calls. A workflow chained several steps. Any of those can fail while the model performs perfectly on the text it was given.

What to remember from this diagram: the dotted arrows all point backwards and none of them is labelled. An end-to-end score tells you something is wrong and cannot tell you what. That is the entire argument for component-level evaluation.

What to measure in a RAG system

Component	The question	What a failure looks like
Retrieval	Did the right chunks come back?	The answer is confidently wrong because the correct passage was never retrieved
Generation	Is the answer faithful to what was retrieved?	The right passage was retrieved and the model contradicted or embellished it

These two demand opposite fixes, and the exam tests exactly that. A retrieval failure is fixed by chunking, embeddings, or the index — changing the model does nothing. A faithfulness failure is fixed by prompting or a different model — improving retrieval does nothing. A team measuring only end-to-end accuracy cannot tell which one they have.

What to measure in agents and workflows

Component	The question
Tool selection	Did it choose the right tool for the step?
Tool arguments	Did it call that tool with correct parameters?
Orchestration	Did it take the right steps, in a sensible order, and terminate?
End-to-end task success	Did the whole thing accomplish what the user asked?

An agent can select a correct tool and pass it wrong arguments. It can make every individual call correctly and loop without terminating. Step-level correctness and task-level success are different measurements, and a system can pass one while failing the other.

Does It Meet the Business Objective?

Objectives 3.4.3 and 3.4.5 are the chapter's pivot, and they are where the largest number of marks are quietly lost.

Objective 3.4.3 names productivity, user engagement, and task engineering as the categories in which an FM is judged against a business objective. Objective 3.4.5 names three concrete alignment metrics: task completion rate, user satisfaction, and cost per interaction.

What to remember from this diagram: both columns contain real numbers. That is what makes the distractor work. The left column is true and irrelevant to the question being asked.

The three named alignment metrics

Metric	What it measures	What it catches that the others miss
Task completion rate	The proportion of user attempts that actually reached the intended outcome	A system that produces beautiful answers nobody can act on. It is behavioural, not aesthetic
User satisfaction	Whether the people using it find it valuable — surveys, ratings, thumbs, repeat use	A system that technically completes tasks while being unpleasant, slow, or untrustworthy to use
Cost per interaction	What one interaction costs to serve	The success that is not worth having. A model that completes more tasks at four times the cost may have made things worse

Cost per interaction is the one most often omitted, and the exam rewards naming it. It is the only alignment metric that can turn an apparent success into a failure. Chapter 05's token-based pricing is what makes it move, and Chapter 09's cost ladder is what it feeds back into.

Productivity, user engagement, and task engineering

Objective 3.4.3's three named categories are broader lenses:

• Productivity — is the work getting done faster or with less effort? Handling time, throughput, volume deflected from a human queue.
• User engagement — are people choosing to use it? Adoption, repeat use, abandonment. A tool that works and that nobody opens has not met the objective.
• Task engineering — whether the task itself was well framed for a model. Sometimes the honest evaluation finding is that the task was the wrong shape, and no model would have met the objective.

Task engineering is the subtle one. It admits the possibility that the failure is not the model's. A scenario where every technical metric is fine and the outcome is still poor may be describing a task that was never suited to an FM — which is the same judgement Chapter 02 taught under Objective 1.2.2.

Decision Rules and Exam Signals

Rule 1 — every metric scores a proxy. Ask what the number is standing in for, and what it cannot see. This is the chapter in one line.

Rule 2 — route by whether a reference answer exists. It exists → benchmark or overlap metric. It does not → human-in-the-loop or LLM-as-a-judge.

Rule 3 — summaries are ROUGE, translation is BLEU. Recall-oriented versus precision-oriented; if the orientation slips, the task association will still carry you.

Rule 4 — "correct but different words" means BERTScore. Overlap metrics punish paraphrase; embeddings do not.

Rule 5 — open-ended with no reference means LLM-as-a-judge, and the judge inherits the limitations of a foundation model.

Rule 6 — Bedrock Model Evaluation runs both automatic and human workflows. Choosing the service is not choosing the method.

Rule 7 — benchmark performance is evidence about the benchmark. "Scores well, underperforms in production" is the expected behaviour, not a paradox.

Rule 8 — an application is not a model. Evaluate retrieval, generation, tool use and orchestration as separate components, because an end-to-end score cannot localise a fault.

Rule 9 — retrieval failure and generation failure demand opposite fixes. Was the right passage retrieved? If no, the fix is the index. If yes, the fix is the prompt or the model.

Rule 10 — a technical score is never business evidence. Task completion rate, user satisfaction and cost per interaction answer the business question; ROUGE never does.

Rule 11 — name cost per interaction. It is the alignment metric most often left out and the only one that can convert a success into a failure.

Rule 12 — traditional ML metrics are a different objective. Accuracy, precision, recall and F1 belong to Objective 1.3.6 and Chapter 03. They need a labelled test set and one right answer, which generated text does not have.

Distractor Patterns

Pattern	What it looks like	How to defuse it
Benchmark score as business proof	A higher benchmark result offered as evidence the deployment succeeded	Different layer. Ask which of the three layers the stem is asking about
BLEU for summarization	The precision-oriented translation metric applied to a summary task	Match the task shape: summaries are ROUGE
ROUGE for translation	The mirror error	Bilingual Evaluation Understudy — the name carries the task
Overlap metric for paraphrase	ROUGE or BLEU offered where the stem stresses different wording, same meaning	Overlap metrics punish paraphrase by construction. BERTScore scores meaning
Accuracy / precision / recall / F1	Classical ML metrics offered for generated text	No confusion matrix exists for a paragraph. Those are Objective 1.3.6, Chapter 03
"Bedrock Model Evaluation is automatic only"	The service described as excluding human review	It manages both automatic and human evaluation workflows
More human review to fix a scale problem	Human-in-the-loop offered where the stem stresses volume	It does not scale with traffic; that is what LLM-as-a-judge addresses
End-to-end score to localise a RAG fault	Overall accuracy offered as the way to find which stage failed	It cannot. Component-level measurement is the named approach
Swap the model to fix retrieval	A better FM offered where the correct passage was never retrieved	The model never saw it. The fix is chunking, embeddings or the index
Improve retrieval to fix unfaithfulness	Index tuning offered where the right passage was retrieved and contradicted	Retrieval already worked. The failure is generation
User satisfaction alone as success	Satisfaction offered while the stem stresses spend	Cost per interaction is the metric that catches an expensive success
A model can't be evaluated without a labelled dataset	Absence of labels presented as blocking evaluation	Human-in-the-loop and LLM-as-a-judge exist precisely for the unlabelled case

The second, third and fourth rows are the most reliable ways to lose marks in Objective 3.4.2, because all three metrics compare a candidate against a reference and the differences are structural rather than obvious.

Scenario Walkthrough

A logistics company deploys an assistant that answers questions about shipping policy from an internal document set. Before launch, the team benchmarked three foundation models and chose the one with the strongest published scores. After launch: answers read well, but customers frequently receive confident statements that contradict the policy documents. Investigation shows that in most failing cases, the passage containing the correct policy was never among the retrieved chunks. Agent handover volume is unchanged from before launch. Finance notes the assistant costs roughly three times per conversation what the team modelled.

Requirement	Reading	Decision
Chose the model on published benchmark scores	Evidence about the benchmark, not about this workload	Benchmarks rank models; they do not predict production. Not a wrong step — an incomplete one
Confident answers contradicting the documents	Sounds like a model or veracity problem	Do not stop here — the next line reclassifies it
Correct passage never retrieved	The model never saw the policy it contradicted	Retrieval failure, not generation. Fix the index, chunking or embeddings — a better FM changes nothing
Handover volume unchanged	The business outcome that justified the build has not moved	Task completion rate is flat — the objective is unmet regardless of any technical score
Three times the modelled cost per conversation	The spend side of the alignment question	Cost per interaction — the metric that turns "it works" into "it is not worth it"

Five requirements, and the third one is the test. A candidate who has decided "this is a hallucination question" at line two will reach for grounding, output validation, or a stronger model — all of them Domain 5 answers to a Domain 3 problem. The stem then supplies the disqualifying fact: the passage was never retrieved. The model cannot contradict a document it was never shown.

The fourth and fifth rows are the pivot the whole chapter exists for. Every technical judgement could have been correct and the deployment would still have failed the business objective, because nothing moved in the only two numbers the business was watching.

Key Concepts

Term	Definition
Human-in-the-loop evaluation	People reviewing and scoring model outputs; the authoritative source for subjective quality, limited by cost, speed and sample coverage
Benchmark dataset	A standardised task set with known good answers, used to score and compare models repeatably; evidence about the benchmark, not about your workload
Amazon Bedrock Model Evaluation	The AWS service for evaluating models, running both automatic metric-based evaluation and managed human evaluation workflows
ROUGE	Recall-Oriented Understudy for Gisting Evaluation — recall-oriented overlap against a reference; the metric family for summarization
BLEU	Bilingual Evaluation Understudy — precision-oriented overlap against a reference; the metric family for translation
BERTScore	A metric comparing embeddings rather than matching words, scoring semantic similarity; the answer when correct output is worded differently from the reference
LLM-as-a-judge	Using a foundation model to score another model's output against stated criteria; scales judgement to open-ended tasks, and inherits a model's biases
Component-level evaluation	Measuring retrieval, generation, tool selection and orchestration separately rather than only end-to-end, so a failure can be localised
Retrieval failure	The correct source passage was never returned; fixed at the index, chunking or embedding layer, never by changing the model
Faithfulness failure	The correct passage was retrieved and the generated answer contradicted or embellished it; fixed at the prompt or model layer
Task completion rate	The proportion of user attempts that reached the intended outcome; a behavioural business metric
User satisfaction	Whether users find the system valuable — ratings, surveys, repeat use, abandonment
Cost per interaction	What one interaction costs to serve; the alignment metric that can turn an apparent success into a failure
Task engineering	Whether the task itself was well framed for a foundation model; the honest finding that a task may be the wrong shape for any model

Revision Flashcards

Say the answer aloud before revealing it.

1. State the one structural fact this whole chapter follows from. → Every metric here scores a proxy, never the thing you actually care about. ROUGE counts overlap with a reference, not usefulness; a benchmark scores a curated dataset, not your traffic; a human reviewer scores the sample they were shown. The skill being tested is knowing which proxy you are holding and what it structurally cannot see.

2. Name the three approaches in Objective 3.4.1 and the relationship between them. → Human-in-the-loop evaluation, benchmark datasets, and Amazon Bedrock Model Evaluation. The first two are methods; the third is the AWS service that runs both — automatic metric-based evaluation and managed human evaluation workflows. Choosing the service does not choose the method.

3. What does a strong benchmark score actually tell you? → That the model performs well on the benchmark. A benchmark is somebody else's curated data; it has never seen your documents, your users' phrasing, or your edge cases. A scenario describing a model that benchmarks well and underperforms in production is not a paradox — it is exactly what benchmarks do and do not measure.

4. ROUGE or BLEU for a summarization task, and why? → ROUGE. It is Recall-Oriented Understudy for Gisting Evaluation, and recall is the natural orientation for a summary: of the material in the reference, how much did the candidate keep? BLEU is precision-oriented and its home task is translation — the Bilingual in the name carries it.

5. Output is correct but phrased entirely differently from the reference. Which metric? → BERTScore. ROUGE and BLEU both count n-gram overlap, so a correct paraphrase scores badly by construction — "the flight was cancelled" and "they called off the flight" share almost no words and all of their meaning. BERTScore compares embeddings, scoring semantic similarity instead of matching words.

6. When is LLM-as-a-judge the right approach, and what is its weakness? → When the task is open-ended and no reference answer exists, and human review will not scale to the volume. Its weakness is structural: the judge is itself a foundation model, carrying the same capacity for bias, inconsistency and confident error as the model it is scoring.

7. Why is veracity the hardest thing for these metrics to catch? → Because a confident hallucination is fluent, well-formed, and often close in embedding space to the truth. Overlap and similarity metrics reward exactly those properties. This is why hallucination detection is treated as its own discipline in Domain 5 rather than as a metric in this objective.

8. Why does an FM application need evaluation separate from the FM? → Because by the time a user sees an answer, the model was one component among several — retrieval, generation, tool selection, orchestration. Any of those can fail while the model performs perfectly on the text it was handed. An end-to-end score tells you something is wrong and cannot tell you what.

9. A RAG system answers confidently and wrongly. What is the first thing to check? → Whether the correct passage was retrieved at all. If it was not, this is a retrieval failure and the fix lives in chunking, embeddings or the index — a better foundation model changes nothing, because it never saw the passage. If it was retrieved and contradicted, this is a faithfulness failure and the fix is the prompt or the model. The two demand opposite responses.

10. Name the three business objective alignment metrics in Objective 3.4.5. → Task completion rate, user satisfaction, and cost per interaction. Completion rate is behavioural — did attempts reach the intended outcome. Satisfaction is whether users find it valuable. Cost per interaction is what one interaction costs to serve.

11. Which alignment metric is most often omitted, and why does it matter? → Cost per interaction. It is the only one that can turn an apparent success into a failure: a system that completes more tasks at several times the cost may have made things worse. Token-based pricing is what makes it move, and it feeds directly back into the customization cost ladder.

12. What does "task engineering" admit that the other categories do not? → That the failure may not be the model's. Sometimes the honest evaluation finding is that the task was never well shaped for a foundation model, and no model would have met the objective — the same judgement Chapter 02 taught about recognising when AI is not the appropriate solution.

The Five-Beat Answer

The core question this chapter prepares you for: "How would you know whether a foundation model is good enough — and whether it was worth deploying?"

Five beats, checked in this order. Missing a beat is a failure state — you will be probed on whichever one you skipped.

1. Approach — say how you would generate a judgement at all: benchmark datasets where reference answers exist, human-in-the-loop where quality is subjective, LLM-as-a-judge where judgement must scale, all runnable through Amazon Bedrock Model Evaluation.
2. Metric — pick from the task shape. ROUGE for summarization, BLEU for translation, BERTScore when correct output is worded differently, LLM-as-a-judge for open-ended work. Say what your chosen metric is blind to, because that is the sentence that shows you understand it.
3. Application — state that the model is not the system. Evaluate retrieval, generation, tool use and orchestration separately, and explain that an end-to-end score cannot localise a fault.
4. Business — move to the layer that authorised the spend: task completion rate, user satisfaction, cost per interaction. Say explicitly that a benchmark score is not evidence here.
5. Judgement — close on the honest possibility: the model may be fine and the task badly framed. Naming task engineering shows you can distinguish a model failure from a problem-selection failure.

A strong answer moves from technical to business evidence and says what each measurement cannot see. A weak answer names ROUGE and stops.

Why This Helps You

On the job: the most expensive pattern in this area is a team that ships on benchmark scores and then cannot explain, six months later, whether the system is working. The second is a RAG deployment where every reported failure is met with "let's try a better model" because nobody ever measured retrieval separately — a fix that cannot work, applied repeatedly, at increasing cost.

In interviews: "how would you evaluate this?" is a standard senior screening question, and most candidates answer with a metric. The strong answer names the layer first, admits what the metric cannot see, and reaches business alignment without being prompted. Being able to say "the model may be fine and the task badly framed" marks out someone who has actually run one of these programmes.

On the exam: Domain 3 is 28% of scored content and this chapter closes it. The highest-value habits are routing a metric from the task shape, localising an application failure to a component, and refusing to accept a technical number as an answer to a business question.

Chapter Checklist

• I can state why every metric in this chapter measures a proxy, and give an example of what one cannot see
• I can name the three approaches in Objective 3.4.1 and say which of them are methods and which is a service
• I can explain what Amazon Bedrock Model Evaluation runs, and why "automatic only" is wrong
• I can say what a strong benchmark score is and is not evidence of
• I can expand ROUGE and BLEU, and attach each to its home task
• I can say why an overlap metric punishes a correct paraphrase, and name the metric that does not
• I can say when LLM-as-a-judge is appropriate and what it inherits from being a model
• I can explain why veracity is the hardest property for these metrics to detect
• I can argue why an FM application needs its own evaluation separate from the model
• I can tell a retrieval failure from a faithfulness failure, and give the opposite fix each demands
• I can name the four things worth measuring in an agent or workflow
• I can name the three business objective alignment metrics and what each catches that the others miss
• I can explain why cost per interaction is the one most often omitted
• I can distinguish these metrics from accuracy, precision, recall and F1, and say which objective those belong to

After the Chapter

1. Complete student/project.md — parts 43-45 of the AI/ML Decision Sheet you began in Chapter 01. Bring the same sheet; do not start a new one.
2. Take student/quiz.md closed-book, then review the reasoning for every question you guessed, including the ones you got right. Pay particular attention to questions 8 and 9 — they are deliberate mirror images, and missing both means the retrieval-versus-generation split has not landed.
3. Open the official v1.1 exam guide's Domain 3 page and confirm you can attach a concept from this chapter to each of the five bullets under Task Statement 3.4. Note that the fifth bullet is about business objective alignment metrics specifically — read it alongside the third, because the exam treats them as a pair.
4. Domain 3 is now complete. Before moving on, check that you can place all four of its task statements: design considerations (Ch 08-09), prompt engineering (Ch 10), training and fine-tuning (Ch 11), and evaluation (this chapter). It is 28% of scored content and the largest single block of marks on the exam.
5. Next: Chapter 13 — Responsible AI: Features, Guardrails, and Legal Risk (Domain 4, Task 4.1, objectives 1-4). This chapter measured whether a model is good. Chapter 13 asks a different question — whether it is acceptable — and a model can pass everything here while failing that one.