Rubrics and Evals Are Not Correct from Day One. Grow Them with Failure Logs

To stabilize AI output, it is useful to build a loop of generation, evaluation, feedback, and regeneration. The idea is straightforward. But once I actually tried to implement it, I got stuck in a different place.

I am running the loop, but I cannot tell what actually improved.

At first, I thought the problem was the prompt or the model. But after reducing the structure into a minimal implementation, the issue turned out to be the evaluation criteria themselves. If the rubric is too coarse, weak output can still pass. If the rubric is too strict, output that is sufficient for the real task can fail unnecessarily.

This article continues the loop engineering series by looking at rubrics and evals not as “perfect judges from day one,” but as operational assets that grow through failure logs. I do not call an external LLM API here. Instead, I use a small verification script prepared for this article to compare a coarse rubric with a more specific one.

LLM Lab When You Build a Minimal API Loop, You Stop Designing Prompts and Start Designing Stop Conditions A minimal generate-evaluate-feedback-regenerate loop that verifies both successful completion and stopping at the maximum attempt count. https://llm-lab.dev/posts/llm-loop-engineering-minimal-api/

The task in this article is generating a first reply to a customer inquiry. Imagine a short response that a support person sends after receiving a report about an outage or defect. For this kind of reply, politeness alone is not enough. The recipient needs to know what to expect next, who owns the situation, and whether the current direction is clear.

That is why this article starts from coarse criteria such as “be polite” and “mention the next response,” then refines them into more specific criteria: “state the conclusion first,” “list at least two next actions,” and “specify the response owner.” The point is not that this rubric is universally correct. The point is to show how evaluation criteria should be grown in response to the task.

Coarse Rubrics Let Weak Output Pass

As an example, consider a task that generates the first reply to a customer inquiry. Suppose we start with the following evaluation criteria:

Be polite
Mention the next response

At a glance, this looks understandable. But as an eval, it is fragile. It does not define what “polite” means, how many next actions are needed, who is responsible, or when the next decision can be made.

With this coarse rubric, even an output like the following can pass:

Thank you for your inquiry.
We will check the situation and respond as needed.

The text is not broken. But as a practical customer support reply, it is weak. It does not state the conclusion first, the next action is vague, and the owner is unclear. Even so, the coarse rubric can still judge it as “polite” and “mentioning a response.”

This is the painful part. The eval is green, but the output is still not usable enough.

Make the Rubric Concrete Enough to Operate

In this verification, I evaluated the same output with two rubrics: a coarse one and a more specific one. The specific rubric checks the following three items:

Does it state the conclusion first?
Does it list at least two next actions?
Does it specify the response owner?

These items are not about beautiful writing. They are minimum conditions for whether the person receiving the inquiry can explain what happens next. The important point is to avoid stopping at vague impressions such as “good writing,” and instead break the criteria down into units that can become feedback when unmet.

The verification script evaluates the same output against two rubrics. It is a custom script prepared for this article, not a standard CLI. It does not call an external API or model.

The script takes a fixed sample reply as input and returns the judgment results for both the coarse and specific rubrics as JSON.

npm run verify

The result looks like this:

{
  "loose": {
    "passed": true,
    "failedRubricIds": []
  },
  "strict": {
    "passed": false,
    "failedRubricIds": [
      "conclusion_first",
      "two_actions",
      "owner_named"
    ]
  }
}

What matters here is not deciding once and for all which rubric is correct. The important observation is that the same output can pass or fail depending on the granularity of the evaluation criteria. In other words, eval failures are not only model failures. They can also be evaluation-design failures.

Failure Logs Become Material for Improving the Rubric

Trying to make a rubric correct from the beginning quickly becomes heavy. I also found myself trying to polish evaluation items too much before the experiment had even moved forward.

In practice, the first rubric can be a hypothesis. What matters is keeping the outputs that passed but were still hard to use, the outputs that failed but were acceptable in practice, and the outputs that repeatedly failed on the same item. Those logs become material for revising the evaluation criteria.

In this example, after seeing a weak output pass the coarse rubric, we can update the criteria as follows:

• Do not treat “be polite” as a standalone pass condition
• Treat “next response” as at least two concrete actions
• Split out “who owns the response” as a separate item
• Add “conclusion comes first” as a condition for faster judgment

This is not prompt improvement. It is eval improvement. Before asking the AI to “be more specific,” the evaluation side needs to define what counts as “specific.”

Do Not Make Green Evals the Goal

This does not mean the eval should simply become stricter.

If you add too many evaluation items, the output starts bending toward the checklist. If every inquiry reply is required to include detailed ownership, deadlines, risks, alternatives, and escalation rules, even a short reply becomes excessive. In a real workflow, you also need to stop at the level of quality the task actually requires.

When growing a rubric, it helps to separate at least these three cases:

• Output that passed but was still hard to use
• Output that failed but was acceptable in practice
• Output that failed and produced concrete feedback for the next run

The third case is especially important. In loop design, a fail judgment alone is not enough. Unless the system can return which item was unmet and what should change next, regeneration is just another retry.

A Minimal Process for Growing Rubrics

Based on this verification, a practical process for growing rubrics and evals looks like this:

1. Start with a coarse rubric of around three items
2. Keep one output that passed but was hard to use
3. Break down why it was hard to use into judgeable items
4. Re-evaluate past outputs with the new items
5. Remove overly strict items, or downgrade them from required conditions to warnings

This process avoids over-designing evaluation criteria at the desk and instead updates them from actual failures. Personally, I have started to see rubrics less as specifications and more as test cases that grow through operation.

Summary

In an LLM output improvement loop, looking only at the generator is not enough to stabilize quality. If the evaluation criteria are too coarse, weak output passes. If they are too strict, output that is sufficient for the task fails.

That is why rubrics and evals should not be treated as things that are correct from the beginning. They are better treated as things that grow through failure logs. Outputs that passed but were unusable, outputs that failed but were acceptable, and feedback that actually improved regeneration all help turn a loop from mere retries into an operational improvement unit.

If this is later connected to observability, the final output alone is not enough. You need to keep the rubric version, unmet items, feedback, and re-evaluation result. Only then can you explain not just that “the AI got better,” but which evaluation criteria improved and what changed.