---
title: "Before Adding a Loop, Decide Which Tasks Should Not Be Looped"
description: "I use API cost, latency, evaluability, and deterministic alternatives as pre-implementation gates, then apply them to fixed scenarios to decide which tasks should not enter a generation-evaluation loop."
lang: "en"
canonical: "https://llm-lab.dev/en/posts/llm-loop-engineering-task-selection/"
source: "https://llm-lab.dev/en/posts/llm-loop-engineering-task-selection.md"
publishedAt: "2026-07-18"
updatedAt: "2026-07-18"
category: "設計"
tags:
  - "agentops"
  - "eval"
  - "loop-engineering"
  - "cost-control"
  - "latency"
---

# Before Adding a Loop, Decide Which Tasks Should Not Be Looped

import LinkCard from "../../components/LinkCard.astro";

> [!NOTE]
> What this article confirms
>
> I passed five fixed tasks through four gates: whether a deterministic alternative exists, agreement among evaluators, maximum-attempt latency, and the upper bound of monthly cost. Four tasks were rejected, leaving only a first reply to an inquiry as a candidate for a small experiment.
>
> This verification is not a benchmark of a particular LLM API. It uses comparison prices and assumed p95 values to reproduce what should be calculated before implementing a loop and the conditions that should stop a task at the entrance.

The loop engineering series has separated generation, evaluation, feedback, and regeneration so failures can be returned to the next attempt. Once observability is connected, it also becomes possible to trace which evaluation item improved on which attempt.

That creates a tempting assumption: if a loop can improve quality, perhaps it should be applied to more tasks.

A loop does not increase quality alone. It also adds evaluator API calls, regeneration latency, costs proportional to the number of attempts, and maintenance work for stop conditions. Worse, if a task cannot be judged consistently, the loop can never decide that the output has improved enough to stop.

The ability to decide not to build a loop matters as much as the ability to build one. Where should that decision sit before implementation begins?

<LinkCard
  href="https://llm-lab.dev/en/posts/llm-loop-engineering-langfuse-observability/"
  title="The Loop Ran, but I Could Not Tell What Worked"
  description="Part five separates generation and evaluation into Observations and Scores for each attempt, making the improvement path observable."
  siteName="LLM Lab"
  image="/images/posts/llm-loop-engineering-langfuse-observability/loop-trace-report.webp"
/>

## Close the Entrance Before Setting the Maximum Attempt Count

A maximum of three attempts prevents an infinite loop, but three attempts are not automatically reasonable. A bounded loop can still be uneconomical. If every generation is followed by an evaluation, the worst case pays for three generations and three evaluations. At sufficient request volume, even a small unit price becomes significant at the monthly level.

I therefore evaluated the entrance in this order:

1. Can deterministic code reproduce the same result?
2. Can the output be judged with sufficient consistency?
3. Can the user tolerate the wait if every request reaches the maximum attempt count?
4. Will the system stay within budget if maximum-attempt executions continue?

The first gate asks whether an LLM is necessary. If calculations, types, required fields, or regular expressions can always return the same pass or fail result, a conventional program should take priority over a regeneration loop.

The second gate asks whether a stop condition can be defined. Being able to generate an output is different from being able to evaluate it. Conditions such as "on brand," "emotionally engaging," or "appropriate as a management decision" may change from one evaluator to another. Feeding unstable judgments into the next attempt makes the loop change direction rather than improve.

The third and fourth gates examine the time and money the system can afford. I do not use an average attempt count here. Before adoption, there is no observed distribution, so the estimate begins with the upper bound in which every task reaches the maximum.

> [!IMPORTANT]
> The maximum attempt count is not only a safeguard against infinite loops. It is also a budget value that caps API cost and latency.

## Passing Five Fixed Scenarios Through the Same Gates

A list of four conditions does not show which gate will reject a real task. I therefore fixed five tasks as inputs: invoice reconciliation, search suggestions, brand copy, nightly batch summaries, and first replies to inquiries. No external LLM API was called. For each task, I supplied monthly volume, maximum attempts, generation and evaluation cost per attempt, assumed generation and evaluation p95 values, tolerated latency, and evaluator agreement.

`npm run verify` is a custom verification command prepared for this article. It feeds the five fixed tasks into the decision gates, asserts that each rejected task produces the expected reason, and generates JSON and HTML comparison reports. It is neither an official CLI nor an evaluation command from a particular provider.

The latency and cost upper bounds use these minimal calculations:

```js
const maximumLatencyEstimate = maxAttempts *
  (generationP95Ms + evaluationP95Ms);

const maximumMonthlyCost = monthlyVolume * maxAttempts *
  (generationCostPerAttempt + evaluationCostPerAttempt);
```

Adding generation p95 and evaluation p95 does not produce a measured end-to-end p95. It is a conservative pre-implementation estimate that combines slow values for the components. The cost estimate also assumes every task reaches the maximum attempt count, so it may be higher than the production average.

I set evaluator agreement of 80% or more as the passing condition. This 80% value is not a general standard; it is a fixed threshold for verifying the decision logic. In production, each task needs a suitable threshold derived from human-labeled samples or results from multiple evaluators.

![Report evaluating five tasks by deterministic alternatives, evaluability, maximum latency, and monthly cost upper bounds](/images/posts/llm-loop-engineering-task-selection/task-selection-report.webp)

Only the first reply to an inquiry passed.

A four-to-one result says nothing about a general adoption rate. It does show that the same decision not to add a loop can have different causes: a deterministic alternative, unacceptable latency, unstable evaluation, or a monthly cost ceiling.

## Do Not Add an Evaluator AI When Code Can Produce the Correct Answer

For invoice reconciliation, I set the maximum latency to 1,340 milliseconds against a tolerated 1,500 milliseconds, the monthly cost upper bound to $24 against a $30 budget, and evaluator agreement to 100%. Looking only at time, cost, and evaluability, the task passes.

I still rejected it. The equality of line-item totals, tax, and invoice totals can be reproduced with formulas and type checks.

Instead of asking an LLM to regenerate and evaluate a calculation, code can determine the result and return an input error when the values do not match. The reason to omit the loop is not poor LLM accuracy. It is that this task does not need probabilistic retries.

## If the User Cannot Wait, the Quality Improvement Never Reaches Them

The fixed conditions for search suggestions use 320 milliseconds for generation, 180 milliseconds for evaluation, and a maximum of two attempts. The estimated maximum latency is 1,000 milliseconds, exceeding the tolerated 450 milliseconds. At 50,000 requests per month, the monthly cost upper bound is also $80 against a $30 budget.

Evaluator agreement is 92%, so a pass or fail decision is possible. That does not make a one-second wait acceptable for every input. Evaluability does not remove synchronous latency.

For this type of task, possible separations include generating once and filtering candidates with safe rules, using a cache, or moving evaluation offline. The entire loop does not need to close within the user's request.

## If Evaluation Is Unstable, There Is No Stop Condition Yet

The brand-copy scenario has enough time and budget, but I set evaluator agreement to 42%. Because it falls below the 80% threshold used in this verification, the task is rejected as not evaluable.

Generating multiple copy candidates can still be useful. If "good" means something different to each evaluator, however, using an evaluation model's pass result as an automatic stop condition cannot establish that the surviving candidate is the best one. More attempts do not solve that problem.

Instead, deterministic constraints such as prohibited phrases and character limits can be checked in code, while a person chooses among the remaining candidates. This does not mean subjective tasks cannot use AI. It means the pass or fail decision for an autonomous loop should not be closed entirely inside AI.

## Even Asynchronous Work Can Be Rejected on Cost at High Volume

The nightly batch summary scenario uses a maximum latency of 7,200 milliseconds against a tolerated 60,000 milliseconds and an evaluator agreement of 91%. Because the result only needs to be ready by the next morning, synchronous latency is not the issue.

At 3,000 tasks per month and up to three attempts, however, the comparison prices produce a monthly upper bound of $108. That is 2.7 times the $40 budget. A task that can afford to be slow is not necessarily cheap.

The entrance needs controls such as sending only likely failures into the loop instead of processing every item, pre-checking the first output with code, or routing work to human review after the monthly budget is reached. Reducing the maximum from three attempts to two is not enough. Unless the number of tasks admitted to the loop is controlled, the volume multiplier remains.

## Passing All Four Conditions Is Still Not Production Approval

The first-reply scenario evaluates whether the conclusion appears first, whether the response contains at least two next actions, and whether it states who will handle the issue. I set its maximum latency to 2,850 milliseconds against a tolerated 5,000 milliseconds, its monthly cost upper bound to $3.60 against a $20 budget, and evaluator agreement to 88%. There is no deterministic alternative.

Under these fixed conditions, it is the only loop candidate among the five tasks.

Passing the gate does not mean the loop is ready for production. The next step is to use a small amount of real data to observe the first-attempt pass rate, actual attempt count, quality difference, and evaluation errors. If there is no meaningful operational difference between the first and third attempts, the loop provides no benefit even though the gates allow it.

The decision gates narrow the list of tasks that are worth experimenting with. They do not replace experimental results.

## Replace the Four Fixed Values with Observations Before Production

When can a candidate from the fixed scenarios proceed to a production decision? Only after the comparison values have been replaced by measurements. At minimum, an actual task should collect:

- p50 and p95 latency for both generation and evaluation
- Input and output token counts and actual cost per attempt
- The distribution of passing attempt numbers and the rate of reaching the maximum
- Evaluator agreement and errors against human ground-truth labels

With those observations, the estimate can move from the upper bound where every task reaches the maximum to an expected value based on the actual attempt distribution. The attempts, Scores, and stop reasons separated in part five become inputs for cost and latency decisions here.

There is no need to pretend that a precise cost-benefit analysis exists before the values can be measured. If a deterministic alternative exists, evaluation is unstable, tolerated latency is exceeded, or the upper-bound budget is exceeded, the entrance can be closed first.

## Conclusion

In these fixed scenarios, invoice reconciliation returned to deterministic code, search suggestions lost the synchronous loop, brand copy retained human selection, and nightly batch summaries required a narrower input set. Only the first reply to an inquiry proceeds to the next experiment.

The possibility of improving generation quality is not enough to justify looping a task. The task must lack a deterministic replacement, support consistent pass or fail evaluation, and fit within maximum-attempt latency and cost limits. Only after satisfying these four conditions does it become a candidate for a small experiment.

Before setting the maximum attempt count, define the conditions that stop the task at the entrance. The first job in loop design is selecting the target, not selecting the number of retries.