AI Security & Responsible AI

"How do you make an AI system secure and responsible — what are the biggest threats and how do you address them?"

The security follow-up to Session 9's production-design question. Interviewers ask this to check whether you think past "does it work" to "is it safe to operate" — a signal of senior-level thinking.

Miss one and they probe.

1. Threat surface — name the four AI-specific attacks by name, distinct from SQL injection/XSS
2. Defence layers — defence in depth, controls at every layer, no single fix
3. Responsible AI — fairness, transparency, accountability, privacy as design constraints, not a checklist
4. Bias & hallucination — both structural, not patchable with a better model
5. Governance — risk register, named owner, escalation path

"AI systems have a threat surface traditional API security doesn't cover — prompt injection, data poisoning, model inversion, and adversarial inputs are all unique to AI. The defence strategy is the same one we use in any complex system: defence in depth — validate and sanitise inputs at the gateway, harden the system prompt at the LLM layer, filter outputs before they reach users, log everything. Responsible AI means building fairness, transparency, accountability, and privacy in from the start. Bias enters through training data and reward signals and needs ongoing demographic auditing. Hallucination is structural — it can't be patched — so for high-stakes decisions I'd always have a human-in-the-loop checkpoint. I'd ship with an AI risk register, a named owner, and a documented escalation path, so when something goes wrong, we know exactly who owns it and what to do."

All five beats, in order — that's what separates this from a vague "we'd add some guardrails" answer.

• "What about security?" — the standard follow-up the moment you finish a system-design answer
• "What happens when the model is wrong?" — probes whether you've thought about accountability, not just accuracy
• "How would you catch bias before launch?" — a direct test of whether Responsible AI is real to you or just a buzzword
• Naming the four AI-specific threats by name signals you've moved past general security into AI-specific security

• A chatbot reveals its system prompt when asked cleverly → prompt injection, needs system-prompt hardening + input sanitisation together
• An HR screening tool quietly favors certain candidate profiles → bias entering through training data, needs a demographic parity audit
• A medical or financial AI feature gives a wrong recommendation with no one accountable → missing HITL checkpoint and missing named owner
• A RAG index gets manipulated by injected documents → data poisoning via an uncontrolled retrieval source

Session 9 built the system that works. This session asks whether it's safe to operate — a different question. Traditional web security controls (parameterized queries, CSRF tokens) don't cover AI-specific attacks, because those attacks exploit natural language, not a structured grammar.

That's why a single control is never enough — it's why defence in depth exists.

Bias and hallucination are the two failure modes that get the most interview attention, because both are structural — properties of how the model is built and trained, not bugs to patch. Bias enters through training data, reward signals, or evaluation sets. Hallucination is what happens when an autoregressive model optimizes for plausibility, not truth.

You can't fix either with a smarter prompt — you can only detect, monitor, and add human checkpoints where the stakes are highest.

An attacker embeds instructions in user input to hijack the model's behavior — ignore its system prompt, reveal internal configuration, or perform unintended actions.

Unlike SQL injection, there's no grammar to parameterize — natural language has no fixed structure to sanitize against, which is why no single filter can fully catch it.

Malicious data injected into training sets or retrieval indices (RAG stores), teaching the model harmful behaviors that activate on specific triggers.

Like poisoning a well upstream — the damage shows up downstream, in every answer that draws from the tainted source, long after the injection happened.

An attacker systematically queries a model to reconstruct data it memorized during training — potentially recovering PII, passwords, or confidential text.

Like reconstructing a shredded document one strip at a time, by asking the shredder the right sequence of questions.

Carefully crafted inputs that cause a model to misclassify, hallucinate, or behave unintentionally on demand — often used to bypass safety classifiers.

Like an optical illusion built specifically to fool one type of eye — it doesn't fool a human, but it reliably fools the model.

Independent security controls at every layer, so bypassing one doesn't compromise the whole: sanitisation at the gateway, prompt hardening at the LLM, output filtering, audit logging at observability.

Like a building with a locked gate, a locked door, and an alarm — each control assumes the previous one might already have failed.

A systematic tendency to favor or disfavor certain groups, entering through training data, reward signals, or evaluation criteria — detected via demographic parity audits and segment-level analysis.

Like a scale that was calibrated on one group's weights — it looks accurate until someone from a different group steps on it.

Confident, fluent, plausible-sounding text that is factually wrong — a structural property of autoregressive models predicting probable tokens, not accurate ones.

Like a very confident storyteller who fills gaps in a half-remembered story with details that sound right — not lying, just optimizing for a good story over a true one.

A human reviews or approves model outputs before they drive a high-stakes action — the only mitigation providing a genuine independent safety backstop for hallucination.

Like a second signature required on a large bank transfer — a deliberate checkpoint precisely because the automated system alone can't be fully trusted for this decision.

Self-assessment, not graded. Answer before checking your notes.

1. Which of the following is an AI-specific threat with no direct equivalent in traditional API security?

• Cross-site scripting (XSS)
• Prompt injection
• Man-in-the-middle attack
• Denial of service

2. A developer says: "We'll just add an output filter to catch harmful responses." What's the primary problem?

• Output filters are too slow, adding unacceptable latency
• A single control is a single point of failure — bypassing it exposes the whole system

Self-assessment, not graded.

3. An AI-powered HR screening tool is being built. Which Responsible AI action should happen before production?

• Publishing a detailed model card explaining training methodology
• Running a demographic parity audit to check for systematic bias across candidate groups

4. Which statement about hallucination is most accurate?

• It's a bug in poorly-trained models that will disappear in future versions
• It's a structural property of autoregressive models and cannot be fully eliminated

Self-assessment, not graded.

5. Which mitigation provides the strongest safety backstop for hallucination in high-stakes decisions?

• Retrieval-Augmented Generation (RAG)
• Human-in-the-loop checkpoints before actions are taken

6. What does the "Accountability" pillar of Responsible AI require in practice?

• Publishing open-source model weights for community audit
• A named individual or team responsible for the system's behavior, with a documented escalation path

Self-assessment, not graded.

7. A chatbot was manipulated via prompt injection into revealing its system prompt. Which two controls would have best reduced this risk — input sanitisation at the gateway, or system-prompt hardening designed so its revelation is non-harmful? (Select both, and say why increasing the context window would not have helped.)

8. Interview-style: an AI medical assistant provides symptom assessments. What single governance artifact matters most before launch, and why?

≈1 hr · no API key needed

Goal: apply a security lens to a real system design, not just recite threat names.

Take a 5-layer AI system design (your own, or the Session 9 production design). For each STRIDE category — Spoofing, Tampering, Repudiation, Information Disclosure, Denial of Service, Elevation of Privilege — identify at least one plausible AI-specific threat, in a table with columns: STRIDE Category, Layer, Threat Description, Why This Matters.

Write down: your completed table (6+ rows) and your top 3 risks with one sentence each on why they ranked highest.

≈45 min · no API key needed

Goal: practice the exact answer format interviewers expect — a technical control paired with a governance control, not just one or the other.

For each of your top 3 risks from Assignment 1, write one technical control (a concrete, implementable engineering measure — specific enough that a developer could build it from your description) and one governance control (a process, policy, or human checkpoint).

Write down: your 3-row mitigation table, plus one sentence per risk on how the technical and governance controls work together.