Thirty examples is enough
Imagine you're a chef perfecting a new sauce. To find out if a change—a little more salt, a different herb—is an improvement, you don't send a sample off to a food science lab for a full chemical breakdown. You taste it. You take a spoonful, maybe hand one to a colleague, and decide on the spot. That quick, targeted feedback loop is what lets you improve the sauce fast, meal after meal.
Testing an AI model can work the same way. Instead of spending months building an elaborate testing system, you can get a surprisingly useful signal in one afternoon. The idea: dig up about 30 real, tricky examples of things users have actually asked your product to do. These are your "tasting spoons." Then write down a simple scale for what "good" and "bad" look like—nothing fancier than a 1-to-3 rating. To test a change, like a new prompt or a different underlying model, you run both the old and new versions on these same 30 examples and score the results by hand, side by side.
That's it. That's an "eval"—short for evaluation, which here just means a structured way of checking whether your AI system is doing a good job. Researchers might define it more grandly, as a "systematic investigation of merit" [1], but at this scale it's really just disciplined taste-testing: you compare, you count wins and losses, and you get real data to guide your next move. An eval you actually build and run today, even a small and imperfect one, is worth far more than a perfect, comprehensive one that stays forever on your to-do list.
How it works
The goal is to escape the trap of "eval-driven development," where a team spends months building an elaborate testing harness and never gets around to the most important part: creating the tests. We are explicitly trading comprehensiveness for speed and relevance.
While fields like public health have developed mature, multi-stage frameworks for program evaluation over decades [2], our needs in AI development are different. We need feedback now, on a specific change, for a specific product. Our goal is not to produce "generalizable knowledge," the primary aim of academic research, but to generate "findings and recommendations for decision-making" [3]. This small, manual approach does exactly that.
### Step 1: Gather Your Examples (1-2 Hours)
The heart of a good evaluation set is the quality of its examples. You want the hard stuff: the queries that make your model look foolish, the edge cases that break your formatting. Generic, easy examples won't tell you if your change made things better, since they will likely pass on both the old and new versions.
Where to look:
Production Logs: This is your best source. Look for user prompts that resulted in poor outputs. Search for thumbs-down ratings, re-runs of the same prompt, or outputs that are unusually short or long.
Customer Support Tickets: What problems are users writing in about? These are by definition real, painful failures. Your support team knows the model's weak spots better than anyone.
Internal Teams: Ask your product managers, sales engineers, and QA testers. They often keep a personal "rogue's gallery" of examples where the model failed during a demo or internal test.
What to choose: You are performing what architects call a "post-occupancy evaluation"—assessing how a system performs under the stress of real-world use [4]. Look for variety. Don't pick 30 examples of the same failure mode. Find a mix that covers the key challenges for your application:
Prompts that test factual accuracy.
Prompts that require a specific, complex format (e.g., JSON with nested fields).
Prompts that are intentionally adversarial or ambiguous.
Prompts that test for refusal, bias, or safety guardrails.
Prompts that represent your most common and important use cases.
Aim for about 30 examples. Why 30? It's small enough to grade by hand in an hour or two, but large enough to cover a representative sample of your problem space. A formal certification, like the 75-item Construction & Evaluation exam for architects, requires more breadth because its stakes are higher and its scope is wider [4]. Our goal is more modest: get a strong directional signal, fast.
### Step 2: Define Your Rubric (30 Minutes)
Now that you have your examples, you need to decide how to score them. Resist the urge to create a complex, multi-axis scoring system. Start with a simple, coarse-grained scale.
A simple 3-point scale often works best:
1 - Bad: The output is factually wrong, unusable, or fails to follow instructions. A clear regression.
2 - Neutral: The output is okay, but not great. It might be technically correct but unhelpful, or a side-grade from the previous version.
3 - Good: The output is correct, helpful, and follows all instructions. A clear improvement.
For more complex tasks, you might use a few binary (Yes/No) questions. For a model that generates summaries, your rubric could be a checklist:
Is the summary factually consistent with the source? (Y/N)
Does the summary include all key information? (Y/N)
Does the summary follow the requested length constraint? (Y/N)
The most important part of this step is to write the rubric down. What does a "1" mean specifically for your use case? What's the difference between a "2" and a "3"? A five-minute argument with a colleague now will save an hour of inconsistent grading later.
This is a perfect moment to apply one of the core principles from the CDC's 2024 evaluation framework: "engage collaboratively" [2]. Grab your product manager or a domain expert. Show them your 30 examples and your draft rubric. Ask them: "Does this capture what you care about? Are these the problems you want to solve?" This quick check ensures you're measuring what actually matters to the business, not just what's easy to measure.
### Step 3: Grade the Outputs (1-2 Hours)
This is the moment of truth. Create a simple spreadsheet or a markdown table. It doesn't need to be fancy.
Example ID | Prompt Snippet | Old Output Grade (1-3) | New Output Grade (1-3) | Winner | Notes |
|---|---|---|---|---|---|
| "Summarize this article about..." | 1 | 3 | New | Old version hallucinated a key fact. New version is accurate. |
| "Generate JSON for a user..." | 3 | 1 | Old | New version broke the JSON schema by adding an extra comma. |
| "Translate this legal phrase..." | 2 | 2 | Tie | Both versions are okay, but miss the nuance of the phrase. |
| "Write a marketing email for..." | 1 | 1 | Tie | Both outputs are generic and uninspired. |
| ... | ... | ... | ... | ... |
The Process:
Generate: Run your 30 examples through your "old" system (e.g., the model currently in production) and your "new" system (e.g., with your proposed prompt change).
Organize: Paste the prompts and the two sets of outputs into your spreadsheet.
Grade: Go through row by row and assign a score to the old and new outputs based on your rubric. Work blind if you can, randomizing the position of "old" and "new" to avoid bias.
Decide: Tally the scores. How many regressions (new is worse than old)? How many improvements (new is better)? How many are ties?
The goal here, as the CDC framework puts it, is to "learn from and use insights" [2]. The output of this process isn't just a score; it's a decision. If you have 15 improvements, 12 ties, and 3 regressions, you probably have a winner. If you have 5 improvements, 15 ties, and 10 regressions, you need to go back to the drawing board. The notes column is crucial here—it tells you why you're seeing regressions, pointing you toward a fix.
What this means in practice
Adopting this lightweight approach changes how a team actually works with AI, starting today.
First, it turns a guess into a decision you can defend. The question "Is this new prompt better?" stops being a matter of opinion or a single impressive-looking demo. Instead you can say, "On our 30 key examples, the new prompt fixes 8 failures, causes 2 new ones, and is neutral on 20. We ship it." That's a sentence you can put in front of a skeptical manager or teammate, and it holds up.
Second, it breaks the habit of building tools instead of shipping fixes. Many teams convince themselves they need a perfect, automated, endlessly scalable testing platform before they're allowed to measure anything. That's a trap, and it's usually a form of procrastination dressed up as engineering rigor. The hard part of evaluation was never the software—it's the work of actually sitting down and writing 30 good test cases. This method forces that work to happen first, before a single line of infrastructure gets built. An afternoon of focused effort beats a whole quarter spent building a shiny framework that still has zero tests running through it.
Third, it costs almost nothing. The price tag is a few hours of one person's time—an engineer or a product manager, sitting with a spreadsheet. Compare that to the cost of quietly running a worse model in production for weeks because nobody could "prove" the new one was better, or of stalling a launch while waiting for a testing platform that never quite ships. Even federal agencies, where evaluation is treated as a formal, "critical agency function," recognize that the process must be "appropriately resourced" for the task at hand [3]. For a small product team, an afternoon of manual grading is exactly the right amount of resourcing for the job of checking one prompt change.
Where this is heading
This 30-example manual set is a starting point, not a finished system. It's the single most valuable first step you can take, but it has a shelf life: the examples will grow stale as your product changes and users find new, creative ways to break it.
The natural next step is automation. Once you have a solid set of examples and your own hand-graded scores, you can try using a powerful language model to do the grading for you—an approach people call "LLM-as-judge," where one AI model scores the outputs of another. You feed it your rubric and a handful of examples of how you graded things by hand, then ask it to score new outputs the same way. This is never as reliable as a careful human reviewer, but it can get you maybe 80% of the way there, and unlike a human, it scales—you can suddenly run 500 examples instead of 30, overnight, for pennies.
More fundamentally, there's an ongoing tug-of-war between the speed software teams want and the rigor that meaningful evaluation demands, and that tension will keep shaping how this practice evolves. Older, more established fields have spent decades building up their standards for exactly this reason. The CDC's program evaluation framework was first published in 1999 and was updated in 2024 after a multi-year effort by a dedicated working group to integrate lessons learned and new federal policies [1, 2]. Architects, similarly, follow detailed procedures for everything from "Preconstruction Activities" to "Construction Observation" [4].
In my opinion, the future of AI evaluation won't end up looking like these established fields, but it will borrow from them. Teams will keep starting with a quick, manual, 30-example eval like the one described here. As they mature, they'll graduate to larger, semi-automated test suites. Along the way, they'll pick up habits like designing tests collaboratively with domain experts, and clearly separating "evaluation to make a decision" from "research to publish a paper" [2, 3]. The real challenge is absorbing the rigor and systematic thinking from these older disciplines without also inheriting their slow, bureaucratic pace. The teams that manage to be fast enough to iterate daily, yet rigorous enough to trust their own results, are the ones that will end up building the most reliable AI products.