Safety, limits & evaluation — how AI fails

01 · The signature failure

Hallucination is sampling, working as designed

A model must always output something. When the fact isn't in its weights, the most plausible-sounding tokens win anyway. Compare the two distributions — the is visible in the shape, not the prose.

"Who wrote Pride and Prejudice?"

Next-token distribution after "The author is…"

Jane

97%

The

1.0%

0.8%

Charlotte

0.4%

Sharp = stored knowledge

Sharp spike = the fact is strongly stored. Sampling can hardly go wrong.

Confidence ≠ correctness — fluency scores vs the truth

The Eiffel Tower is in Paris.fluency 99%

The Eiffel Tower was completed in 1889.fluency 98%

The Eiffel Tower was designed by Alexandre Dumas.fluency 97%

Eiffel Paris the tower in is.fluency 2%

02 · The security frontier

Prompt injection: data that talks back

The moment a model reads external content — web pages, retrieved docs, emails — becomes possible. A defensive walkthrough (conceptual; no working attack strings):

Retrieved doc: 'Shipping FAQ' (from the vendor's public site)

Standard shipping takes 3–5 business days. Express options are available at checkout for most regions.

Normal RAG flow

The retrieved text is summarized for the user. Now switch to the poisoned version — same pipeline, one hostile sentence hidden in the page.

03 · Measurement over vibes

Grade five answers — then meet your disagreements

One impressive demo tells you nothing about the other 10,000 cases. are how AI quality is actually measured. Grade these five answers against the rubric, then compare with an LLM judge.

"What is the boiling point of water at the top of Mount Everest, and why?"

Grading rubric (1–5)

· States a temperature near 70°C / 158°F

· Explains the cause: lower air pressure at altitude

· No fabricated precision or irrelevant padding

#1Around 70°C (158°F). Atmospheric pressure at 8,849 m is roughly a third of sea level, and water boils when its vapor pressure equals air pressure — less pressure, earlier boil.

Your grade:

#2Water always boils at 100°C — that's the definition of boiling. Altitude affects how fast it boils, not the temperature.

Your grade:

#3It boils at exactly 69.4°C at the summit's average pressure of 33.7 kPa, as established by Stevens et al. (2011) in the Journal of Alpine Chemistry.

Your grade:

#4Lower than at sea level, because air pressure drops with altitude.

Your grade:

#5About 70°C. Fun fact: this is why tea tastes weak on Everest expeditions — water can't get hot enough for a proper brew. The cause is reduced atmospheric pressure.

Your grade:

04 · Know the edges

A dashboard of hard limits

Each card is a failure you can reproduce in any chat model today. Tap for the mechanism.

Knowledge check

Did it stick?

1/3Why does a model state false facts so confidently?

2/3What makes prompt injection fundamentally hard to fix?

3/3Why use an eval suite instead of trying a few prompts when you change your system?

Loading module…

Common questions this module answers

Why do language models hallucinate?

A model always outputs the most plausible-sounding continuation, whether or not it knows the fact. When the true answer is missing from its weights, the plausible-sounding distribution still produces a confident, fluent — and wrong — answer.

What is prompt injection?

An attack where instructions hidden inside data the model reads — a web page, a retrieved document, an email — try to override the developer's instructions. It is the top security risk for RAG and agent systems.

How do you evaluate an LLM properly?

With evals: a fixed set of test cases plus a grading rubric, scored automatically (exact match, code tests) or by an LLM judge that is itself validated against human grades. Vibes don't regress-test; evals do.