Fine-Tuning vs Prompt Engineering: How to Pick the Right Lever

Fine-tuning is expensive, slow, and often unnecessary.

That’s not an opinion — it’s what you discover after spending a week preparing a training dataset, running a fine-tuning job, evaluating the results, and realising a well-crafted 5-shot prompt would have got you 90% of the way there in two hours.

The good news: the decision is mostly mechanical. Ask three questions. The answers tell you where to invest.

First, what each lever actually changes

The two get compared as if they were rival tools for the same job. They aren’t.

• Prompt engineering changes what the model reads on each call: the instructions, the examples, the format you ask for. Nothing about the model itself moves. If you delete the prompt, the behaviour is gone.
• Fine-tuning changes the model’s weights by training it on hundreds of input-output pairs. The behaviour is baked in, so the prompt can get shorter — but every change to that behaviour means training again.

That one difference drives everything else in this post: prompting is cheap to change and costs you on every call; fine-tuning is expensive to change and can cost you less on every call. Which one wins depends on how settled your task is and how often you run it.

There is also a third lever that people mistake for fine-tuning: retrieval (RAG), which looks up the right documents at question time and puts them in the prompt. Keep it in mind — it is the answer to Question 2 more often than fine-tuning is.

The 3 questions before you fine-tune

Question 1: Have you tried prompt engineering with at least 5 examples?

Few-shot prompting — giving the model 3 to 10 examples of the input-output pattern you want — dramatically narrows model behaviour. If you haven’t done this, you’re not ready to fine-tune. Full stop.

Most tasks where “the model gets it wrong” are actually tasks where the prompt doesn’t show the model what “right” looks like in your specific domain.

There is a second half to this question: can you tell whether a change helped? OpenAI’s own fine-tuning guide puts it bluntly — “only invest in fine-tuning after setting up evals” (OpenAI, supervised fine-tuning). Anthropic’s prompting guide asks for the same thing before you even start prompting: clear success criteria and a way to test against them (Anthropic, prompt engineering overview). A fixed set of test cases with known right answers is the minimum. Without it, you can’t know whether the fine-tuned model is better or just different. (How to tell a prompt change actually helped walks through building one.)

Question 2: Is the gap a knowledge gap or a behaviour gap?

A knowledge gap means the model doesn’t know facts it needs — internal product names, recent events, proprietary data. Fine-tuning doesn’t reliably solve this. RAG does.

A behaviour gap means the model knows the facts but formats the output wrong, uses the wrong tone, or fails to follow a consistent structure you need. That’s a format and pattern problem — something few-shot prompting addresses directly.

Fine-tuning genuinely wins when you have a style or task pattern that’s too complex to demonstrate in a context window — or when you need to consistently reproduce a very specific output schema across thousands of calls.

Question 3: What are the real costs?

The next section puts real numbers on the cost row.

What it costs, with real numbers

Data. OpenAI’s fine-tuning API accepts as few as 10 examples, but its guide says it sees improvements from 50–100 and recommends starting with 50 well-crafted demonstrations (OpenAI, supervised fine-tuning). Those are the floor for a hosted fine-tune, not a promise. Every example also has to be correct: a fine-tuned model learns your mistakes as faithfully as your good answers, so the real cost is the hours someone spends checking labels.

Per-call price. This is where the usual intuition — “fine-tuned means cheaper” — needs checking against a price list. Take OpenAI’s gpt-4.1-mini as one example, using the prices on OpenAI’s pricing page when we checked on 6 October 2026, per million tokens:

The fine-tuned model costs twice as much per token. It only comes out cheaper per call if the examples you no longer have to send outweigh that. Work it through for a classifier where the customer’s message is 200 tokens and the answer is 10 tokens:

• Few-shot prompt with 600 tokens of examples: 800 input tokens × $0.40 + 10 output × $1.60 ≈ $0.34 per 1,000 calls.
• Fine-tuned, no examples: 200 input × $0.80 + 10 output × $3.20 ≈ $0.19 per 1,000 calls.
• Few-shot prompt with only 100 tokens of examples: 300 × $0.40 + 10 × $1.60 ≈ $0.14 per 1,000 calls — now the prompt is cheaper.

With these prices, the fine-tuned model is cheaper per call only when your examples take up more tokens than the message plus four times the answer. Prices change and other providers price differently, so redo this sum with your own token counts and today’s price list. Two more things can move it: prompt caching (OpenAI lists cached input for base gpt-4.1-mini at $0.10 per million, a quarter of the normal price, on the same page), which makes a long, unchanging example block much cheaper; and moving to a smaller model, which a fine-tune sometimes makes possible.

The cost nobody puts in the table: the fine-tuned model is a thing you now own. When the label set changes, when the provider retires the base model, when a new model would do the job better out of the box — each one is a retrain and a re-evaluation. A prompt change is an edit and a test run.

When prompting is the right choice

Reach for a better prompt first, and stay there, when:

• The task is still changing. New categories, a new output field, a policy that gets revised every month. Each change is a prompt edit, not a training run.
• You have fewer than about 50 good labelled examples. That is below where OpenAI says it starts to see fine-tuning improvements. Put your best ones in the prompt instead.
• The problem is missing facts. Use retrieval to put the facts in the prompt. Training facts into weights is unreliable and goes stale.
• The problem is output shape. If the answer must be valid JSON with set fields, check whether your provider has a structured output mode first. OpenAI’s Structured Outputs, for example, “ensures the model will always generate responses that adhere to your supplied JSON Schema” (OpenAI, Structured Outputs). That solves a problem many teams once fine-tuned for.
• Volume is modest. If the extra example tokens cost a few dollars a month, no amount of fine-tuning will pay back the engineering time.

When fine-tuning is the right choice

Fine-tuning earns its cost when several of these are true together — one alone is rarely enough:

• You have done the prompting work — many genuinely different prompts, not one — and your test set shows a gap that stays put.
• The gap is behaviour, not knowledge: a house style, a tone, a specialised labelling scheme the model keeps drifting from even with examples.
• The task is stable. The labels and the output format won’t change for months.
• You have hundreds of checked examples, or can produce them.
• You run it at high volume, and the per-call sum above says a shorter prompt (or a smaller model) saves real money.
• The prompt is too long for your latency budget, and removing the examples would fix it.

A decision checklist

Go down this list in order. Stop at the first “no”.

1. Do you have a test set with known right answers, and a score you trust? If no — build that first. Nothing else can be judged without it.
2. Have you tried at least five genuinely different prompts, including 5–10 worked examples and the cases the model gets wrong? If no — prompt.
3. Is the gap about facts the model doesn’t have? If yes — add retrieval, not training.
4. Is the gap about output shape that a structured output mode would enforce? If yes — use that mode.
5. Is the task stable enough that you won’t relabel the data in the next few months? If no — keep prompting.
6. Do you have at least 50 checked examples, and ideally hundreds? If no — keep prompting while you collect them.
7. Does the per-call sum, at your real volume and today’s prices, favour the fine-tuned model — or is latency the binding limit? If no — the prompt is the cheaper system.
8. Does the fine-tuned model beat your best prompt on the test set by a clear margin? If yes — fine-tune. If not — ship the prompt.

Four worked examples

A support-ticket classifier with four categories. A 5-shot prompt like the one in the benchmark below usually gets most of the way. Fine-tune only if your test set shows a stubborn error pattern and you run enough tickets for the per-call sum to matter.

An assistant that answers questions about the company’s leave policy. This is a knowledge gap. Fine-tuning would bake in this year’s policy and quietly keep answering with it after the policy changes. Retrieve the current policy document and put it in the prompt.

Extracting invoice fields into JSON. This looks like a fine-tuning job because it is about format, but a structured output mode enforces the shape directly. Use that, and spend your examples on the hard part: the fields the model misreads.

Rewriting product descriptions in a strict house style, at high volume. This is the case fine-tuning was made for. The style is hard to capture in a few examples, the task doesn’t change, you can collect hundreds of approved before-and-after pairs from past work, and the volume makes a shorter prompt worth something. Even here, start with the prompt, so you have a score for the fine-tuned model to beat.

Few-shot prompting patterns that actually work

The structure that consistently produces clean, reproducible outputs:

You are [specific role]. You [precise task description].

Format: [exact output format]

Examples:
---
Input: [example 1 input]
Output: [example 1 output]
---
Input: [example 2 input]
Output: [example 2 output]
---
[3–5 more examples]
---
Input: {user_input}
Output:
Copy

Three things that make few-shot prompts work:

1. Consistency in examples — every example must follow the same format, including edge cases. Mixed formatting in examples produces mixed output.
2. Cover the failure cases — if the model keeps producing output X when you want Y, add an example that shows the X-input → Y-output mapping explicitly.
3. End the prompt with Output: and nothing else — this cues the model to continue the pattern, not explain it.

When fine-tuning wins: LoRA basics

If prompting genuinely isn’t enough and you are training an open model yourself, LoRA (Low-Rank Adaptation) is the practical way to do it. Instead of updating all model weights, LoRA injects small trainable adapter matrices into the model’s layers. The original model weights freeze; only the adapters train.

Why this matters for you:

• Far fewer weights to train. The LoRA paper reports that, compared with fully fine-tuning GPT-3 175B, it cut the number of trainable parameters by 10,000 times and the GPU memory needed by 3 times (Hu et al., LoRA).
• The adapter is a separate, small file. You keep one copy of the base model and swap adapters per task.
• The tooling is mature. Hugging Face’s peft library wraps a model with LoRA adapters, and TRL’s SFTTrainer accepts a LoRA config through its peft_config argument, so a training run is a few lines (TRL, SFT Trainer).

What a run costs in GPU time depends on the model size, the data and the hardware, so price it from your cloud provider’s current GPU rates rather than from anyone’s rule of thumb — including ours.

End project: benchmark few-shot vs LoRA on the same task

This script runs a classification task twice — once with 5-shot prompting, once with a LoRA-adapted model — and prints a comparison table. It is the last step of the checklist above, made concrete.

#!/usr/bin/env python3
"""
Benchmark: 5-shot prompting vs LoRA-adapted model
Requirements: pip install transformers peft datasets torch
"""

import time
from transformers import AutoTokenizer, AutoModelForCausalLM, pipeline
from peft import PeftModel
import torch

# --- Config ---
BASE_MODEL = "mistralai/Mistral-7B-Instruct-v0.2"
LORA_ADAPTER_PATH = "./lora-adapter"   # path to your trained LoRA adapter
DEVICE = "cuda" if torch.cuda.is_available() else "cpu"

# Test dataset: (input, expected_output) pairs
TEST_CASES = [
    ("The shipment arrived damaged.", "complaint"),
    ("Can you tell me your business hours?", "inquiry"),
    ("I'd like to cancel my subscription.", "cancellation"),
    ("This is the best product I've ever used!", "praise"),
    ("My order hasn't arrived in 3 weeks.", "complaint"),
]

# Few-shot prompt template
FEW_SHOT_PROMPT = """Classify the customer message into one of: complaint, inquiry, praise, cancellation.

Examples:
---
Message: "The item broke after one use."
Category: complaint
---
Message: "What payment methods do you accept?"
Category: inquiry
---
Message: "Amazing quality, will buy again!"
Category: praise
---
Message: "Please cancel my order #12345."
Category: cancellation
---
Message: "I never received my package."
Category: complaint
---
Message: "{input}"
Category:"""


def first_word_after_category(output):
    words = output.split("Category:")[-1].strip().split()
    return words[0].lower() if words else ""


def run_few_shot(test_cases, model_id, device):
    tokenizer = AutoTokenizer.from_pretrained(model_id)
    model = AutoModelForCausalLM.from_pretrained(
        model_id, torch_dtype=torch.float16, device_map=device
    )
    pipe = pipeline("text-generation", model=model, tokenizer=tokenizer, max_new_tokens=10)

    results = []
    for text, expected in test_cases:
        prompt = FEW_SHOT_PROMPT.format(input=text)
        start = time.time()
        output = pipe(prompt)[0]["generated_text"]
        latency = time.time() - start
        predicted = first_word_after_category(output)
        results.append({
            "input": text[:40],
            "expected": expected,
            "predicted": predicted,
            "correct": predicted == expected,
            "latency_ms": round(latency * 1000),
            "method": "5-shot"
        })
    return results


def run_lora(test_cases, base_model_id, adapter_path, device):
    tokenizer = AutoTokenizer.from_pretrained(base_model_id)
    base = AutoModelForCausalLM.from_pretrained(
        base_model_id, torch_dtype=torch.float16, device_map=device
    )
    model = PeftModel.from_pretrained(base, adapter_path)
    pipe = pipeline("text-generation", model=model, tokenizer=tokenizer, max_new_tokens=10)

    results = []
    for text, expected in test_cases:
        # LoRA model uses minimal prompt — no examples needed
        prompt = f"Classify this customer message: \"{text}\"\nCategory:"
        start = time.time()
        output = pipe(prompt)[0]["generated_text"]
        latency = time.time() - start
        predicted = first_word_after_category(output)
        results.append({
            "input": text[:40],
            "expected": expected,
            "predicted": predicted,
            "correct": predicted == expected,
            "latency_ms": round(latency * 1000),
            "method": "LoRA"
        })
    return results


def print_table(results):
    print(f"\n{'Method':<8} {'Input':<42} {'Expected':<14} {'Predicted':<14} {'Correct':<8} {'Latency'}")
    print("-" * 100)
    for r in results:
        tick = "YES" if r["correct"] else "NO"
        print(f"{r['method']:<8} {r['input']:<42} {r['expected']:<14} {r['predicted']:<14} {tick:<8} {r['latency_ms']}ms")

    for method in ["5-shot", "LoRA"]:
        subset = [r for r in results if r["method"] == method]
        accuracy = sum(r["correct"] for r in subset) / len(subset) * 100
        avg_latency = sum(r["latency_ms"] for r in subset) / len(subset)
        print(f"\n{method}: accuracy={accuracy:.0f}%  avg_latency={avg_latency:.0f}ms")


if __name__ == "__main__":
    print("Running 5-shot prompting benchmark...")
    few_shot_results = run_few_shot(TEST_CASES, BASE_MODEL, DEVICE)

    print("Running LoRA benchmark...")
    lora_results = run_lora(TEST_CASES, BASE_MODEL, LORA_ADAPTER_PATH, DEVICE)

    print_table(few_shot_results + lora_results)
Copy

To train the LoRA adapter before running this benchmark, use TRL’s SFTTrainer with a LoraConfig passed as peft_config, on a few hundred labelled examples that are not in the test set. Five test cases are enough to see the script work; to make a real decision, grow the test set to a few hundred cases drawn from real traffic. The point of the script isn’t to show one always wins — it’s to make the tradeoff concrete and measurable on your actual task.

If the accuracy numbers are within a few points of each other, ship the prompt. If the fine-tuned model wins clearly on a task you run at high volume, and the per-call sum agrees, the training investment can pay off. Now you have data to make that call.

Key takeaways

• Prompt engineering changes what the model reads on each call; fine-tuning changes the model’s weights. Prompts are cheap to change, fine-tunes are expensive to change.
• Build a test set with known right answers before either. Without one you can’t tell whether a fine-tuned model is better or just different.
• Try several genuinely different prompts with 5–10 worked examples before considering fine-tuning.
• If the model lacks facts, use retrieval (RAG), not fine-tuning. If the output shape is wrong, try a structured output mode first.
• Hosted fine-tuning needs labelled data: OpenAI accepts as few as 10 examples but recommends starting with 50 well-crafted ones.
• A fine-tuned model is not automatically cheaper per call: OpenAI’s fine-tuned gpt-4.1-mini costs twice the base price per token, so it only wins when the examples you stop sending outweigh that.
• Fine-tune when the gap is behaviour, the task is stable, you have hundreds of checked examples, volume is high, and the fine-tuned model beats your best prompt on the test set.

See this applied in the ML path →