The Loop Ran, but I Could Not Tell What Worked

Once loop design is translated into code, output quality can be handled as a flow of generation, evaluation, feedback, and regeneration rather than as a one-shot prompt. That much had become clear in the previous articles.

But a different discomfort appeared when I started looking at the execution logs.

The loop is running. But I cannot tell what worked.

If I only look at the final output, I can judge whether the reply satisfied the three evaluation conditions. But that is not enough for operations. I also want to know which attempt stopped the loop, which rubric item remained unresolved until the end, whether the evaluation criteria should change, and whether the generation prompt should change. For that judgment, logs for each attempt are necessary.

What became clear is that keeping only “whether the final output satisfied the three criteria” is insufficient for observing a loop. To use the result later for improvement decisions, one loop execution needs to be treated as a Trace, each generation and evaluation step as an Observation, and each rubric result as a Score. In this fifth article in the loop engineering series, I used a verification script prepared for the article to create a fixed loop execution and sent that payload to Langfuse. This is not about creating more visually impressive logs. It is about sending data at a granularity that can support later improvement decisions.

LLM Lab Rubrics and Evals Cannot Be Correct from the Start, so I Grow Them with Failure Logs A verification article on comparing a loose rubric with a more concrete one, and treating Evals as something that grows from failure logs. https://llm-lab.dev/posts/llm-loop-engineering-rubric-eval/ LLM Lab Building a Minimal API Loop Turns Prompt Improvement into Stop-Condition Design A verification article on implementing a minimal loop with separated generation and evaluation, including both successful completion and max-attempt stopping paths. https://llm-lab.dev/posts/llm-loop-engineering-minimal-api/

Final Output Alone Does Not Make a Loop Observable

In the third article, I built a minimal loop that separates generation and evaluation and always stops at a maximum number of attempts. In the fourth article, I confirmed that if the rubric is too loose, a weak output can still be treated as satisfying the evaluation conditions.

When those two pieces are connected, the target of observation changes slightly. Keeping only the completed text does not show where the loop improved.

• Which attempt it was
• What feedback was passed into generation
• Which rubric items were still unmet during evaluation
• Which rubric version was used
• Why the loop stopped
• How the final output should be handled

If the transition of unmet rubric items is dropped, the loop becomes almost a black box. For example, suppose three items fail on the first attempt, only one remains on the second attempt, and all three are satisfied on the third attempt. In that case, the point is not that the prompt generally improved. The point is to trace which evaluation items disappeared because of feedback.

“The AI improved” is too coarse. Only when you can see that “the owner was the only thing it could not write until the end” can you decide the next improvement target.

Decide the Langfuse Granularity First

Langfuse’s Observability Overview explains Application Tracing for tracking LLM application requests, prompts, responses, token usage, latency, and tool execution. Its Evaluation Overview also organizes how to continuously evaluate LLM application behavior using live traces, datasets, experiments, and Scores.

Given that premise, the first design decision for loop engineering is not whether to send data to Langfuse. It is what should become a Trace, what should become an Observation, and what should become a Score. If the whole loop is sent as one large log, the person reading it later still cannot tell where it failed.

For the payload, I separated the units as follows.

Observation Unit	What It Represents
Trace	One full loop execution for one task
Observation	Generation and evaluation for each attempt
Score	Pass/fail result for each rubric item
Metadata	Attempt number, stop reason, rubric version, and output hash

In loop design, I want to know which feedback actually contributed to regeneration. For that reason, separating Scores by rubric item is easier to work with.

Sending the Loop to Langfuse with a Verification Script

I used a custom verification script prepared for this article. It is not a Langfuse official CLI. The script takes a fixed inquiry-reply task as input, performs mock generation, rubric evaluation, feedback, and up to three regeneration attempts. It then converts the result into JSON that can be sent as Langfuse Traces, Observations, and Scores.

No external LLM API is called. The generated outputs are mocked. For Langfuse, however, the public key, secret key, and Base URL can be provided through environment variables, and actual sending is enabled only when explicitly requested. By default, the script runs as a dry run and produces only JSON and an HTML report, so the shape of the payload can be checked even in an environment without credentials.

The actual sending follows this approach. Trace and Observation data are sent together to the Langfuse ingestion API, and Scores are sent to the Score API. To avoid duplicate Scores appearing on the same Trace during reruns, the script adds a runId only for actual sending and separates traceId, observationId, and scoreId. This is a small detail, but it matters for reproducibility in a verification article. If the same traceId is sent repeatedly, later inspection mixes up “how many loops happened” with “how many times the verification script was run.”

The JSON generated by the verification script roughly follows this shape.

Internal Label	Reader-Facing Meaning
rubric.conclusion_first	Whether the conclusion comes first
rubric.two_actions	Whether there are at least two next actions
rubric.owner_named	Whether it says who will handle the issue

{
  "trace": {
    "name": "support-reply-loop",
    "metadata": {
      "rubricVersion": "support-reply-rubric-v2",
      "maxAttempts": 3,
      "stopReason": "passed",
      "rawContentStored": false
    }
  },
  "observations": [
    { "type": "generation", "name": "attempt-1.generate" },
    { "type": "span", "name": "attempt-1.evaluate" }
  ],
  "scores": [
    { "name": "rubric.conclusion_first", "value": 0 },
    { "name": "rubric.two_actions", "value": 0 },
    { "name": "rubric.owner_named", "value": 0 }
  ]
}

rawContentStored: false is included as a marker that generated text is not retained in the public report. In real operations, whether prompts and output text should be stored in Langfuse depends on the nature of the data being handled. For this article’s report, I kept output hashes, character counts, unmet items, and stop reasons instead of the generated body text itself.

Sending to Real Langfuse Made the Improvement Path Visible

The actual send produced 1 Trace, 6 Observations, and 9 Scores. Each of the three attempts has both a generation Observation and an evaluation Observation, so there are 6 Observations. The rubric has 3 items, so across 3 attempts there are 9 Scores.

After sending, I read back the Trace name, Observation count, and Score count through the Langfuse Public API. I confirmed 6 Observations by traceId, and when filtering the Score list by the current traceId, I confirmed 9 Scores. One small snag was that if I looked at the Score list result as-is, Scores from previously sent traces were mixed in. At first, it looked like there were more than 9. Right, I was looking at the same list.

This confirmation changed what I could say in the article. It is no longer just that the dry-run structure was correct. I could confirm that even after sending to Langfuse, the decomposition into Trace, Observations, and Scores remained intact, and that the improvement path by attempt could be traced.

The point of this report is not only that the final output satisfied all three criteria. In the Score history, all three items fail on the first attempt, only “whether it says who will handle the issue” remains unmet on the second attempt, and all items pass on the third attempt.

In this loop, “put the conclusion first” and “write at least two next actions” were resolved by the second attempt. Only “write who will handle the issue” remained until the end. Once this is visible, the next improvement targets become much more concrete.

• Add “write who will handle the issue” to the generation prompt from the beginning.
• Make the feedback more specific when the owner is missing.
• Change who should be named depending on the inquiry type.
• Fall back to human confirmation when the owner is unknown.

These do not come from a log that only says “the third attempt satisfied all three criteria.” In loop observability, the important thing is not just that the loop succeeded, but the order in which failures disappeared before success.

Not Everything Should Go into Langfuse

Increasing observability logs can feel reassuring. But logs for LLM applications may contain user input, generated output, business context, and evaluation comments. Before sending them to an observability platform such as Langfuse, you need to decide what to store and what not to store.

For this payload, I set a constraint that the full generated body text would not be included. Instead, I kept the output hash, character count, attempt number, and unmet rubric items. Of course, there are cases in real debugging where the body text is necessary. Even then, it is better to separate at least the following decisions.

• Store body text only during development
• Store body text even in production
• Store only masked body text
• Store only evaluation results and metadata
• Delete stored data after a specific number of days

Personally, I felt it was safer to first confirm whether the improvement path can be traced without body text, rather than assuming full-text logging from the beginning. In this verification, even without sending the body text, I could trace which rubric disappeared at which attempt. If that is not enough, expand the stored data later. Do not widen the data surface from the start.

Minimal Checklist for Loop Observability

From this verification, the minimal checklist for observing loops in Langfuse looks like this.

1. Treat one user task or one business process as one Trace.
2. Split generation and evaluation into separate Observations.
3. Put the attempt number into every Observation’s metadata.
4. Split Scores by rubric item.
5. Put rubricVersion into the Trace and evaluation Observations.
6. Put stopReason into Trace metadata.
7. Store max-attempt stopped cases in the same shape as successful cases.
8. Decide whether generated body text will be stored before implementation.
9. Separate runId so Scores do not duplicate during resend or rerun.

With this shape, it becomes a little easier to distinguish “the model is bad,” “the prompt is bad,” and “the evaluation criteria are bad” when looking back later. It does not automatically identify the root cause, but it at least gets you out of the state where you stare only at the final output and wonder what happened.

Running the Loop Is Not Enough

When observing loops with Langfuse, the granularity needs to be designed before increasing the amount of logging. Treat one loop execution as a Trace, split generation and evaluation for each attempt into Observations, and store the pass/fail result for each rubric item as Scores. After actually sending the data to Langfuse, this shape made it easier to trace the improvement path.

If you look only at the final output, the loop ends with “whether it satisfied the three criteria.” But by looking at unmet items for each attempt, you can see which failures disappeared and which failure remained until the end. Only then can you decide whether to fix the prompt, fix the rubric, or add a fallback.

There are also limitations from this verification. Because body text is not stored, it cannot reveal problems in the generated wording or style. Also, Score readback was confirmed by filtering the list by the current traceId, so in real operations the retrieval conditions and aggregation unit need to be designed separately. The remaining design areas are body-text storage, masking, retention period, Score naming conventions, and runId design for resending.

Running the loop is not enough. Only when the loop is designed down to an observable granularity can it become operational.