Tips for Reliably Extracting Structured JSON from LLMs
I summarize the approaches and tips for extracting structured data from llms as reliably, accurately, and with as low latency as possible.
The Challenge of Getting Clean json from llms
When developing applications with llms — especially agents and backend automation pipelines — receiving output as clean structured data (json) is a hard requirement.
However, simply asking the model to “output in json” in the prompt often results in unwanted explanatory text mixed in or subtle schema mismatches that cause parse errors. This is an all-too-common occurrence.
In this post, I summarize the approaches and tips for extracting structured data from llms as reliably, accurately, and with as low latency as possible.
1. Three Approaches to Structured Data Extraction
Currently, there are three main ways to obtain structured data from major llm apis.
| Approach | Characteristics and When to Use |
|---|---|
| Structured Outputs | The strongest option. Since the model output itself is forcibly constrained to a json schema, schema violations are in principle 100% eliminated. |
| json Mode | Constrains the output format to json. However, the prompt must also explicitly say “return in json”, and since key types and structure cannot be fully enforced, post-parse validation (zod, etc.) is mandatory. |
| Tool Calling (Function Calling) | Originally intended for external tool integration, but because structured data can be defined as “arguments to a tool (function)”, it indirectly functions as a reliable means of json extraction. |
💡 Conclusion: If you want to guarantee schema completeness and the model supports it, Structured Outputs (or the Tool Calling hack) is the only choice.
2. Tips for Schema Design and Prompting to Improve Accuracy
Simply using the mechanism (structured outputs, etc.) is not enough. Designing a schema that “does not confuse the llm” is directly linked to suppressing hallucinations and improving extraction accuracy.
① Don’t Skimp on description
When defining objects with json schema or zod, always provide a concrete description for each property (key).
Since llms interpret this description as a strong instruction (prompt), verbalizing “what to extract and by what criteria” here is the most effective approach.
② Reduce Ambiguous Free-Form Fields and Leverage enum
For example, when performing sentiment analysis on text, rather than letting the model freely return a string, restrict the choices with an enum such as ['positive', 'negative', 'neutral'].
This completely prevents variations in phrasing (e.g., “positive”, “high rating”) caused by the llm’s wavering thoughts.
③ Embed a Reasoning Phase (Chain of Thought) in the Schema
When you want the model to go through a thinking process before reaching a conclusion, yet still want the final output in json, a useful technique is to prepare a reasoning field in the schema in advance.
Since llms output json from top to bottom, having the model perform reasoning in reasoning first dramatically improves the accuracy of the final result (conclusion).
// Schema structure example
{
"reasoning": "The thought process leading to the conclusion (write this first)",
"final_answer": "The final extracted result or judgment"
}3. Operational Considerations (From an llmops Perspective)
- Trade-off between token consumption and latency:
Including a reasoning phase (
reasoning) in the json significantly increases the number of output tokens, which in turn slows down the response and raises api costs. For real-time screens, you may need to deliberately omit reasoning or call a lightweight model (such as a flash-class model). - Compatibility when changing schemas: When you change application-side code (zod definitions, etc.), failing to update the schema definition passed to the llm can cause unexpected parse errors in production. It is safest to manage schema logic in a single, centralized location.
Summary
Extracting structured data from llms has completely shifted from prompt-level hacks (messy approaches) to api-level features (robust systems).
By adhering to basics such as writing thorough description fields and fixing values with enum, and by switching between “speed priority” and “accuracy priority” depending on the use case, you can achieve the best results.
References
- [Title and link of the original article, etc.]
