Evaluating LLM apps - notes from Chip Huyen's AI Engineering (chapters 3 and 4)

I've been reading Chip Huyen's book, AI Engineering, published in 2025. I just can't stop reading it. I'm really into understanding the technical side of everything I work with, and the book covers a little bit of all of it. It will take me some time to finish, since it's a bit long. I'm currently on chapters 3 and 4, the ones about evaluation, but I'm already writing this article with my thoughts and how I'm using what I've learned in my real projects, both in development and in production.

My main motivation to read the book was the evaluation part. As she says, evaluating AI applications is a tough process. To make sure that your apps fulfill the role they're supposed to, you need clear methods and metrics, and that's where evaluation comes in. If you don't know what you need to measure, you're probably going to end up with a product that can't prove its value.

The big problem when evaluating foundation models (LLMs, image generators, and so on) is that they're open-ended, so they don't have a closed set of outputs. This makes it much harder to guarantee that the outcomes are what we want for a specific task. In chapter 3, where she lists the challenges of evaluating foundation models, she also points out that the smarter models get, the harder they are to evaluate: most people can spot a first grader's wrong math, but few can do the same with a PhD-level solution. It is easy to know whether the sky is blue, so if the LLM prints out "the sky is red", we know that, in most cases, it is wrong. But when you have domain-specific terms and facts, like in scientific or industrial fields, it is much harder to know whether the output is wrong. One example from my own work: our SDR assistant, which qualifies industrial leads over WhatsApp, offered a juice manufacturer products that our knowledge base lists for biodiesel and dairy plants. The message was fluent and polite, and to anyone outside the industry it looked perfectly fine.

This leads us to several different approaches. Chapter 3 has a whole section on AI as a judge, including its limitations, like inconsistency and ambiguous criteria. One point I keep coming back to: a judge isn't just a model, it's the model plus its prompt, so changing either one gives you a different judge. I've been combining LLM-as-a-judge with human evaluation, until I can trust the judge most of the time. I build a dataset from cases we've already seen in production, from real conversations to previous interactions with our AI assistant, and then I set up experiments with metrics that represent whether the outputs fulfill the task. For example, the SDR needs to qualify leads and get relevant information from them while keeping a natural persona. We also need to take the cost per message into account, since this app's volume in production is high.

With the experiment results, we can keep adjusting the architecture, the prompts and the nodes, and switching models, until we can see that we're getting closer to our goal: the metrics we set and the intention behind each output. Adding domain experts to the loop is really useful too, but I found it important to align the criteria of the LLM judges and the human judges. This way, we can compare them and tune the AI judges to approximate the human specialist. In chapter 4, she calls creating a clear evaluation guideline "the most important step of the evaluation pipeline", and part of that step is validating your rubric with humans and refining it until they can follow it without ambiguity. Our assistant first plans a goal for each turn and then writes the message. In my first calibration round, I blind-labeled 40 of those turn goals and compared my labels with the judge's. We agreed on only 65% of them (Cohen's kappa 0.26). Reading the disagreements was the most valuable part: some came from rubrics that were too strict, and seven came from business rules nobody had ever written down. After fixing both, agreement on the same 40 items rose to 95% (kappa 0.89). That number is optimistic, since I tuned on those exact items, so the real test is still a new fresh round.

Those rules mattered more than the choice of model. With them, a small model (Claude Haiku) went from 71.3% to 93.8% of goals that fulfill the rubric, matching the large one (Claude Opus, 92.5%) at 1/18 of the cost per call. It's not always easy to get a large amount of data from human evaluation, as it is expensive both in cost and time, so creating a good judge that can stand in for it is important: it lets you run this kind of comparison every time something changes.

Evaluation also measures your own safeguards. We had a reviewer step that checked every message before it was sent. When I measured it, the full reviewer rewrote 21.7% of the drafts, and 12 of its 13 rejections had no severe failure according to the judge. In a few cases, the rewrite even made the message less natural. Sending only the risky drafts to a lighter reviewer, picked by a cheap classifier, meant about 20% of the turns reached the reviewer. The cost per turn dropped from $0.042 to $0.022, with no more severe failures than reviewing every message.

Evaluation also tells you where to fix things. The juice manufacturer's mistake didn't need more review: it came from retrieval, which was returning chunks about the wrong industry. Filtering the knowledge base by the lead's segment took the turns that use retrieval from 94.1% to 100% without severe failures. That's the first step of the evaluation pipeline she proposes in chapter 4: evaluate each component of the system on its own, not only the final output, or you won't know where it fails.

All of these numbers come from a lab, with simulated leads based on real ones, 40 to 81 cases and one or two repetitions per experiment. They're small samples, and the next steps are confirming them against labels from our sales team and, later, in production.

The main lesson so far: an evaluation is only as good as the agreement between your judge and the people who know the business, and building that agreement brings out the rules your product was missing. I'll probably write more reviews related to the book, as there's still a lot to learn from it and to test in my current projects. I'm already writing the case of IARA-SDR-2 as well, the new version of the SDR assistant we have in production, so the full numbers and technical details will show up there.

---

I wrote this article and edited it with the help of AI. The ideas, the experiments and the numbers are mine.