Where Does a Flue Agent Fail? Separating Skill Execution and Structured Output
A Flue 1.0 Beta verification note that compares normal prompting with explicit skill execution on the same issue input, separating finish completion, schema validation, and classification quality.
This article is part of " Flue Practical Guide " series (Part 7) — 7 posts
When I built an issue triage agent with Flue 1.0 Beta, the most memorable failure was that the model seemed to understand finish in natural language, but could not reach the finish tool call that Flue expected.
After that, I sent Flue’s observe(...) events to Langfuse and checked both success and failure cases as traces.
One question still bothered me.
In the issue triage agent, the workflow called session.prompt(..., { result }). If I explicitly call the Flue skill with session.skill('triage', { result }), can I still receive structured output in the same way?
According to Flue’s documentation, prompt(), skill(), and task() can all receive a Valibot schema through options.result. So this is a meaningful API-level comparison. This time, I passed the same issue payload through two routes, session.prompt() and session.skill(), and checked what stays the same and where the evaluation criteria start to diverge.
The Short Version
With this input, both session.prompt(..., { result }) and session.skill('triage', { args, result }) succeeded, and both returned structured data that matched the Valibot schema.
However, the outputs were not identical. severity and reproducible matched, but the granularity of labels and summary differed slightly. I do not read this as a failure. It means that even inside the same schema, model judgment and invocation route can still introduce variation.
The takeaway is not that session.skill() has a problem with structured output. On the contrary, session.skill() can also return structured data. The important point is that agent frameworks need to be evaluated in layers: API connectivity, tool protocol, schema validation, and output quality. The earlier Agent did not call finish failure belongs before schema validation, at the tool protocol layer.
Building a Comparison Workflow
For this verification, I reused the existing issue triage agent and the triage skill, and only added a comparison workflow.
export async function run({ init, payload }: FlueContext<IssuePayload>) {
const harness = await init(issueTriage);
const promptSession = await harness.session('prompt-result');
const skillSession = await harness.session('skill-result');
const promptResult = await captureInvocation('prompt-result', async () => {
const response = await promptSession.prompt(`Use the triage skill to classify this GitHub issue.
Title:
${payload.title}
Body:
${payload.body}
Return only the structured triage result.`, {
result: TriageResult,
});
return response.data;
});
const skillResult = await captureInvocation('skill-result', async () => {
const response = await skillSession.skill('triage', {
args: {
title: payload.title,
body: payload.body,
},
result: TriageResult,
});
return response.data;
});
return {
input: {
title: payload.title,
bodyCharacters: payload.body.length,
},
results: [promptResult, skillResult],
};
}The key point is that I separated prompt-result and skill-result into different sessions. If I ran both operations in the same session, the conversation history from the first run could affect the second run. What I wanted to compare here was the difference between a prompt() route that instructs the agent to use the skill and a skill() route that explicitly invokes the skill, while keeping the same agent, schema, and payload. Separating the session names keeps the comparison narrower.
The schema is the same one used in the existing issue triage workflow.
const TriageResult = v.object({
severity: v.picklist(['low', 'medium', 'high', 'critical']),
reproducible: v.boolean(),
labels: v.array(v.string()),
summary: v.string(),
});The args passed to session.skill() are treated as the skill input. I also added one sentence to the triage skill so that it knows to use args.title and args.body when they are provided.
Read the issue title and body provided in the prompt. If this skill is invoked
with `args.title` and `args.body`, use those values as the issue title and body.Without this, the model may have to infer where the issue body is from the Markdown skill alone. If a skill is meant to be a reusable unit of work, the expected arguments in the skill text should line up with the input contract in code.
What the Verification Command Means
For execution, I used an npm script added for this article. It is not an official Flue command; it is a thin wrapper that internally calls flue run compare-skill-prompt --target node --payload ....
npm run compare:skill-prompt -- '{"title":"Dashboard is blank after login","body":"Steps: log in, open /dashboard. Expected widgets. Actual blank white screen in Chrome 126."}'The input is the issue title and body. The target is the comparison workflow. I checked three things:
- Whether
session.prompt(..., { result })returns schema-valid data as a prompt route instructed to use the triage skill. - Whether
session.skill('triage', { args, result })returns schema-valid data as an explicit triage skill invocation route. - Whether both routes reach Flue’s completion condition,
finish.
Both routes returned success.
{
"input": {
"title": "Dashboard is blank after login",
"bodyCharacters": 90
},
"results": [
{
"route": "prompt-result",
"status": "success",
"data": {
"severity": "high",
"reproducible": true,
"labels": ["bug", "frontend", "dashboard"],
"summary": "ダッシュボードにログイン後、ウィジェットが表示されるはずの画面が真っ白になるという問題。Chrome 126で再現可能とのこと。再現手順は「ログイン後 /dashboard を開く」と明記されており、事象は明確。ただし、コンソールエラーやサーバーログ、他ブラウザでの確認結果などの詳細情報は不足している。"
}
},
{
"route": "skill-result",
"status": "success",
"data": {
"severity": "high",
"reproducible": true,
"labels": ["bug", "dashboard", "ui"],
"summary": "ログイン後にダッシュボード（/dashboard）へアクセスすると、ウィジェットが表示されるべきところが空白の白い画面になってしまう。Chrome 126で発生。"
}
}
]
}What is interesting here is that the results are not identical. They are different within the same schema. prompt-result includes frontend and mentions missing extra information in the summary. skill-result includes ui and gives a shorter summary. Both are valid structured data.
In other words, structured output fixes the shape of the output, but it does not fully fix the judgment inside that shape. If I conflate those two, I may wrongly conclude that “it returned JSON, so evaluation is done.” Whether the output is shaped for downstream processing and whether the classification is good enough are separate evaluation axes.
Which Layer Did the Earlier finish Failure Belong To?
In the earlier failure log, the model mentioned finish. However, it could not return the tool call required by Flue’s execution protocol, and eventually failed with this error:
The agent gave up: Agent did not call `finish` or `give_up` after 33 attempts.Calling this simply a “structured output failure” is too coarse. With this comparison in mind, the layers look like this:
At the API connectivity layer, the provider accepted the request and returned model output. Natural language output existed. But the model did not reach the finish tool call expected by Flue. In that state, it is not yet meaningful to evaluate whether the result matches a Valibot schema. Schema validation only matters after the agent satisfies the completion protocol.
Wait, it looks like it understands in text, but it still cannot cross that boundary?
That uncomfortable gap matters a lot when verifying agent frameworks. With a one-off LLM API call, a plausible natural language answer is often enough to say “it worked.” But with Flue, where skills, tools, workflows, and structured output are combined, the model must do more than write an answer. It has to follow the operation sequence required by the framework.
When I Would Use session.skill
From this experiment, session.skill() looks like a good fit for workflows that explicitly execute a skill. Examples include issue triage, review, summarization, and classification tasks where the same criteria should be called repeatedly.
On the other hand, session.prompt() is useful when I want to assemble the instruction text flexibly each time. In the existing issue triage workflow, the prompt explicitly said, “Classify this issue and return only the structured result.” That form is easy to read and small to implement.
However, if the judgment criteria are managed as a skill, using session.skill('triage', { args, result }) makes the responsibility boundary clearer. The workflow receives input, passes arguments to the skill, and receives schema-backed data. The classification criteria stay in the Markdown skill. That shape should make it easier to revise the skill later or call the same skill from another workflow.
What Should Be Evaluated
After this comparison, I think a Flue issue triage eval should separate at least the following checks:
| Axis | What to Check | Example Failure |
|---|---|---|
| Connectivity | Whether model specifier, auth, and base URL work | Unknown model specifier, 401, connection failure |
| Protocol | Whether the run reaches activate_skill or finish | Agent did not call finish |
| Schema | Whether the output matches the Valibot schema | Missing required fields, type mismatch |
| Quality | Whether severity and labels are appropriate | Schema is valid, but classification is weak |
| Stability | Whether route or model changes break the result for the same payload | Label or summary drift |
The important distinction is that “the model can be swapped” and “the same execution protocol remains stable” are not the same claim. An OpenAI-compatible endpoint can be configured, the response can come back, natural language can make sense, Flue can reach finish, the schema can pass, and the classification can be good enough. Those should be treated as separate stages, not one binary success flag.
Summary
At least for this issue triage case, session.skill() worked fine with structured output. I passed the same payload through session.prompt(..., { result }) and session.skill('triage', { args, result }), and both routes reached finish and returned data matching the Valibot schema.
That does not mean every model will behave stably. The earlier Agent did not call finish failure happened before structured output schema validation, at the Flue tool protocol layer. Once that distinction is clear, model selection and eval design become much easier to reason about.
Personally, I find it more valuable to preserve “which layer failed” than to keep only clean success logs when verifying Flue. An agent is not just a function that returns natural language. It is an execution actor that has to complete an operation sequence inside a framework. So for a framework like Flue, I want to separate five checks: API request succeeded, model returned output, finish was reached, schema validation passed, and the classification was actually good enough.
