Saving Eve Eval Failures to Langfuse Datasets to Improve Evaluation Criteria

When I built a small Eve agent with a weather tool, the first eval failed. The tool output contained condition: "partly cloudy", but the final assistant reply translated that condition into Japanese.

つれづれなる Agent OPS Building a Tool-Calling Agent with Vercel's Eve and Running It from the TUI A follow-up validation log that adds tools and evals to Eve. https://llm-lab.dev/posts/vercel-eve-deep-dive/

Locally, the fix was simple. I stopped checking for partly cloudy in the final reply, asserted it on the tool output via t.calledTool(...), and checked the user-facing reply for strings such as 26°C and the dummy-data note.

From an AgentOps perspective, however, stopping there loses a useful artifact. A failed eval is not just a red mark. It can become a reusable regression case for the next model, prompt, or instruction change. In this note, I send the failure to a real Langfuse Dataset and compare the original failed run with the revised run.

The Tool Did Not Fail

The target eval uses a tiny dummy weather tool. It returns a fixed temperature and condition.

// agent/tools/get_weather.ts
export async function fetchWeather({ city }) {
  return {
    city,
    temperatureC: 26,
    condition: "partly cloudy",
    note: "this is dummy data from a local check, not a real weather API",
  };
}

The failed assertion expected includes("partly cloudy") on the final reply. Eve did call the tool and receive the expected structured output, but the model then produced a natural Japanese sentence.

The lesson is that agent evals should separate internal behavior from final expression. Tool calls and tool outputs can be checked as structured facts. Final replies are user-facing language, so translation, paraphrasing, and summarization may happen.

The corrected eval checks the structured output separately.

// evals/weather.eval.ts
await t.send("東京の天気を教えてください");
t.completed();
t.calledTool("get_weather", {
  output: { city: "Tokyo", condition: "partly cloudy" },
});
t.check(t.reply, includes("26°C"));
t.check(t.reply, includes("ダミー"));

What to Save in Langfuse

Langfuse Evaluation is designed around reusable test cases and experiment comparisons. The Langfuse docs describe Datasets as collections of inputs and expected outputs, and SDK experiments as a way to compare prompts, models, and code variants.

For this Eve failure, the minimum useful split is:

Element	What to save	Why
Dataset item	User input and expected tool output	Re-run the same case later
Trace	Actual tool output and final reply	Inspect what happened in that run
Dataset run item	Which run executed which dataset item	Compare prompt or model changes
Score	tool_output_condition, reply checks, and overall_eval_pass	Separate what passed from what failed

The important part is not to collapse the whole case into pass: false. The tool worked. The flawed part was the evaluator’s expectation that an internal English value would appear unchanged in the final answer.

Sending the Runs to Langfuse

The npm run sync:eval-dataset command used here is not an Eve built-in. It is a verification script for this article. It reads two JSON fixtures, one for the failed run and one for the revised run, and sends Dataset, Dataset item, Trace, Dataset run item, and Score payloads to Langfuse.

Without credentials, it can still run as a dry-run:

npm run sync:eval-dataset -- --dry-run

The two runs are attached to the same Dataset item.

{
  "evalName": "weather",
  "runName": "weather-eval-v1-final-reply-condition-check",
  "input": {
    "message": "東京の天気を教えてください"
  },
  "datasetExpected": {
    "tool": {
      "name": "get_weather",
      "output": {
        "city": "Tokyo",
        "condition": "partly cloudy"
      }
    }
  }
}

The comparison is about evaluation design, not model quality.

Run	Evaluation policy	overall_eval_pass
weather-eval-v1-final-reply-condition-check	Expect partly cloudy in the final reply	0
weather-eval-v2-split-tool-and-reply-checks	Check partly cloudy on tool output and user-facing facts in the final reply	1

That makes the history more useful than a single failed status. It says the tool output matched the expectation, while the final reply string check was the unstable part. The revised run records the improvement as an evaluation-policy change.

Result

To send the payloads, configure Langfuse credentials and explicitly enable syncing.

LANGFUSE_SYNC=1 npm run sync:eval-dataset

This created the eve/weather-eval-regression Dataset, one Dataset item, and two runs. Environment-specific IDs are omitted here.

{
  "ok": true,
  "datasetName": "eve/weather-eval-regression",
  "runs": [
    {
      "runName": "weather-eval-v1-final-reply-condition-check",
      "scores": ["tool_output_condition", "reply_includes_partly_cloudy", "overall_eval_pass"]
    },
    {
      "runName": "weather-eval-v2-split-tool-and-reply-checks",
      "scores": ["tool_output_condition", "reply_includes_temperature", "reply_mentions_dummy_data", "overall_eval_pass"]
    }
  ]
}

In the Langfuse Dataset item, the input message, expected tool output, and final-reply checks are stored together.

The Experiments tab then shows both runs for the same Dataset item. The v1 run has reply_includes_partly_cloudy and overall_eval_pass as False, while the revised v2 run records the split reply checks and overall_eval_pass as True.

Summary

This small Eve eval failure led to three reusable points.

1. The partly cloudy failure was not a tool-call failure. It was an expectation-design failure on the final reply.
2. When storing the case in Langfuse, input, expected tool output, actual output, run, and scores should be separated.
3. Linking the failed v1 run and revised v2 run to the same Dataset item makes the overall_eval_pass change from 0 to 1 explainable as an evaluation-criteria improvement.

The remaining comparison axis is to run another model or instruction variant against the same Dataset item. At that point, Eve’s local eval starts turning into a reusable evaluation history rather than a one-off gate.