LLM evals cheat sheet: what to check, how, and when

This is the one page we wish we had when we started building with language models. It covers what to evaluate, how to check it and when to run the checks.

01 / 111. Start with real cases

An eval set is a list of inputs with a description of what a good output must contain and must not do.

• Take inputs from real work: tickets, emails, documents. Invented prompts miss the hard cases.
• Write expected behaviour in the words of the person who will use the system.
• Include cases where the right answer is to refuse, say "I do not know" or hand to a person.
• Start with about twenty. Add every bug you find.

02 / 112. Check with code where you can

Code checks are fast, cheap and do not drift. Use them for:

• Output format and schema validity.
• Required fields present, forbidden content absent.
• Values in range: dates are dates, amounts are sane, ids exist.
• Tool use: the right tool, in the right order, with valid inputs.
• For retrieval: is the chunk with the answer in the top results?

Rule: if you can check it with code, check it with code.

03 / 113. Use rubrics for judgement

Some qualities need judgement: correctness of an explanation, tone, completeness.

• Write rubric lines that are concrete. "Mentions the approval limit" works. "Is helpful" does not.
• Grade pass or fail per line, with a one line reason.
• Two people grading the same output should agree. If they do not, fix the rubric line.

04 / 114. Model judges, carefully

A model can apply a rubric at scale. Before trusting it:

• Have a person grade a sample, around fifty outputs.
• Run the judge on the same sample.
• Read every disagreement and improve the rubric.
• Use the judge only when it agrees with people on almost all of the sample.
• Re check with a fresh human sample regularly, because inputs change.

05 / 115. Evaluate agents step by step

For agents, the final answer is not enough.

• Check the tool calls: which, in what order, with what inputs.
• Check that actions needing approval were approved.
• Check the agent stopped when it should.
• Keep the trace of each eval run so failures can be replayed.

06 / 116. Set a ship bar

• Agree the overall score that means ship.
• Mark critical cases. A single failure on a critical case blocks release.
• The owner of the system signs this before the build.

07 / 117. When to run evals

Run the full set on every:

• Prompt change, however small.
• Model or model version change.
• Retrieval change: chunking, index, number of results.
• Tool contract change.

Compare with the previous run case by case. A total score that stays the same can hide cases that got worse.

08 / 118. Keep it growing

• Every bug becomes a case.
• Every rejected action at an approval step becomes a case.
• Every production issue becomes a case.

09 / 11Common mistakes

• Evals written after the demo, by the people who built it.
• Only happy path cases.
• A model judge trusted without checking against people.
• Evals run once at launch and never again.

10 / 11Practise with feedback

If you want to build and defend an eval set on a realistic project, the AIXL Academy Forward-Deployed AI Engineering cohort assesses your evals, not only your demo.

Details and enrolment: AIXL Academy Forward-Deployed AI Engineering cohort

11 / 11Sources

• None: AIXL Academy curriculum practice, no external facts or figures.