---
title: "Saving Eve Eval Failures to Langfuse Datasets to Improve Evaluation Criteria"
description: "A validation note on sending a failed Eve weather-tool eval to Langfuse, comparing the failed run with a revised run, and improving the evaluation criteria."
lang: "en"
canonical: "https://llm-lab.dev/en/posts/vercel-eve-eval-langfuse-dataset/"
source: "https://llm-lab.dev/en/posts/vercel-eve-eval-langfuse-dataset.md"
publishedAt: "2026-07-02"
updatedAt: "2026-07-02"
category: "AgentOps"
tags:
  - "vercel"
  - "eve"
  - "langfuse"
  - "eval"
  - "agent"
---

# Saving Eve Eval Failures to Langfuse Datasets to Improve Evaluation Criteria

import LinkCard from "../../components/LinkCard.astro";

> [!NOTE]
> What this article confirms
>
> I took a failed Eve weather-tool eval and saved both the failed run and the revised run to the same Langfuse Dataset item. The failure was not a broken tool call; it was an evaluation-policy mistake that expected `partly cloudy` to appear unchanged in the final reply.
>
> Splitting Dataset item, Trace, Dataset run item, and Score made the change explainable: v1 failed because the final-reply check was unstable, while v2 passed after checking the structured tool output and user-facing reply separately.

When I built a small Eve agent with a weather tool, the first eval failed. The tool output contained `condition: "partly cloudy"`, but the final assistant reply translated that condition into Japanese.

<LinkCard
  href="https://llm-lab.dev/posts/vercel-eve-deep-dive/"
  title="Building a Tool-Calling Agent with Vercel's Eve and Running It from the TUI"
  description="A follow-up validation log that adds tools and evals to Eve."
  siteName="つれづれなる Agent OPS"
  image="/images/posts/vercel-eve-deep-dive/heroImage.webp"
/>

Locally, the fix was simple. I stopped checking for `partly cloudy` in the final reply, asserted it on the tool output via `t.calledTool(...)`, and checked the user-facing reply for strings such as `26°C` and the dummy-data note.

From an AgentOps perspective, however, stopping there loses a useful artifact. A failed eval is not just a red mark. It can become a reusable regression case for the next model, prompt, or instruction change. In this note, I send the failure to a real Langfuse Dataset and compare the original failed run with the revised run.

## The Tool Did Not Fail

The target eval uses a tiny dummy weather tool. It returns a fixed temperature and condition.

```ts
// agent/tools/get_weather.ts
export async function fetchWeather({ city }) {
  return {
    city,
    temperatureC: 26,
    condition: "partly cloudy",
    note: "this is dummy data from a local check, not a real weather API",
  };
}
```

The failed assertion expected `includes("partly cloudy")` on the final reply. Eve did call the tool and receive the expected structured output, but the model then produced a natural Japanese sentence.

![The first Eve eval failed only on the final-reply string check, while the tool call itself passed](/images/posts/vercel-eve-eval-langfuse-dataset/eve-eval-fail.webp)

The lesson is that agent evals should separate internal behavior from final expression. Tool calls and tool outputs can be checked as structured facts. Final replies are user-facing language, so translation, paraphrasing, and summarization may happen.

The corrected eval checks the structured output separately.

```ts
// evals/weather.eval.ts
await t.send("東京の天気を教えてください");
t.completed();
t.calledTool("get_weather", {
  output: { city: "Tokyo", condition: "partly cloudy" },
});
t.check(t.reply, includes("26°C"));
t.check(t.reply, includes("ダミー"));
```

![The Eve eval passed after separating tool-output checks from final-reply checks](/images/posts/vercel-eve-eval-langfuse-dataset/eve-eval-success.webp)

## What to Save in Langfuse

Langfuse Evaluation is designed around reusable test cases and experiment comparisons. The Langfuse docs describe [Datasets](https://langfuse.com/docs/evaluation/experiments/datasets) as collections of inputs and expected outputs, and [SDK experiments](https://langfuse.com/docs/evaluation/experiments/experiments-via-sdk) as a way to compare prompts, models, and code variants.

For this Eve failure, the minimum useful split is:

| Element | What to save | Why |
|---|---|---|
| Dataset item | User input and expected tool output | Re-run the same case later |
| Trace | Actual tool output and final reply | Inspect what happened in that run |
| Dataset run item | Which run executed which dataset item | Compare prompt or model changes |
| Score | `tool_output_condition`, reply checks, and `overall_eval_pass` | Separate what passed from what failed |

The important part is not to collapse the whole case into `pass: false`. The tool worked. The flawed part was the evaluator's expectation that an internal English value would appear unchanged in the final answer.

## Sending the Runs to Langfuse

The `npm run sync:eval-dataset` command used here is not an Eve built-in. It is a verification script for this article. It reads two JSON fixtures, one for the failed run and one for the revised run, and sends Dataset, Dataset item, Trace, Dataset run item, and Score payloads to Langfuse.

Without credentials, it can still run as a dry-run:

```bash
npm run sync:eval-dataset -- --dry-run
```

The two runs are attached to the same Dataset item.

```json
{
  "evalName": "weather",
  "runName": "weather-eval-v1-final-reply-condition-check",
  "input": {
    "message": "東京の天気を教えてください"
  },
  "datasetExpected": {
    "tool": {
      "name": "get_weather",
      "output": {
        "city": "Tokyo",
        "condition": "partly cloudy"
      }
    }
  }
}
```

The comparison is about evaluation design, not model quality.

| Run | Evaluation policy | `overall_eval_pass` |
|---|---|---:|
| `weather-eval-v1-final-reply-condition-check` | Expect `partly cloudy` in the final reply | 0 |
| `weather-eval-v2-split-tool-and-reply-checks` | Check `partly cloudy` on tool output and user-facing facts in the final reply | 1 |

That makes the history more useful than a single failed status. It says the tool output matched the expectation, while the final reply string check was the unstable part. The revised run records the improvement as an evaluation-policy change.

![The Eve eval failure and revised run split into Langfuse Dataset, run, and score payloads](/images/posts/vercel-eve-eval-langfuse-dataset/dataset-payload-dry-run.webp)

## Result

To send the payloads, configure Langfuse credentials and explicitly enable syncing.

```bash
LANGFUSE_SYNC=1 npm run sync:eval-dataset
```

This created the `eve/weather-eval-regression` Dataset, one Dataset item, and two runs. Environment-specific IDs are omitted here.

```json
{
  "ok": true,
  "datasetName": "eve/weather-eval-regression",
  "runs": [
    {
      "runName": "weather-eval-v1-final-reply-condition-check",
      "scores": ["tool_output_condition", "reply_includes_partly_cloudy", "overall_eval_pass"]
    },
    {
      "runName": "weather-eval-v2-split-tool-and-reply-checks",
      "scores": ["tool_output_condition", "reply_includes_temperature", "reply_mentions_dummy_data", "overall_eval_pass"]
    }
  ]
}
```

In the Langfuse Dataset item, the input message, expected tool output, and final-reply checks are stored together.

![A Langfuse Dataset item containing the Eve eval input, expected tool output, and final-reply checks](/images/posts/vercel-eve-eval-langfuse-dataset/langfuse-dataset-item.webp)

The Experiments tab then shows both runs for the same Dataset item. The v1 run has `reply_includes_partly_cloudy` and `overall_eval_pass` as False, while the revised v2 run records the split reply checks and `overall_eval_pass` as True.

![The Langfuse Experiments tab comparing score differences between the failed run and the revised run](/images/posts/vercel-eve-eval-langfuse-dataset/langfuse-experiment-runs.webp)

## Summary

This small Eve eval failure led to three reusable points.

1. The `partly cloudy` failure was not a tool-call failure. It was an expectation-design failure on the final reply.
2. When storing the case in Langfuse, input, expected tool output, actual output, run, and scores should be separated.
3. Linking the failed v1 run and revised v2 run to the same Dataset item makes the `overall_eval_pass` change from 0 to 1 explainable as an evaluation-criteria improvement.

The remaining comparison axis is to run another model or instruction variant against the same Dataset item. At that point, Eve's local eval starts turning into a reusable evaluation history rather than a one-off gate.