Large language models don’t actually follow a format. They generate text by sampling probabilities, one token at a time.

You can write the best prompt imaginable—« Respond only with valid JSON. This is extremely important. »—and the model will still occasionally get it wrong. Not every time, but often enough to break a production application.

As long as LLMs were used for conversations, that wasn’t a big deal. A human could read the response and understand what the model meant. But now that LLMs have become building blocks for software—powering agents, tools, and function calling—they’re no longer talking to humans. They’re talking to code. And code is unforgiving: one extra comma, and JSON.parse() fails.

Fortunately, there’s a way to make invalid JSON impossible, not just unlikely. Let’s see how it works, and why it made such a difference in a real-world project: Mission Radar AI.

Why Prompting Alone Is Never Enough

An LLM generates text one token at a time. At each step, it assigns a score to every possible token in its vocabulary—these raw scores are called logits. Modern models typically have vocabularies of around 100,000 to 130,000 tokens, meaning they compute that many scores for every generation step.

Those logits are then passed through a function called softmax, which converts them into probabilities between 0 and 1 that sum to 1. The model samples the next token from this probability distribution—essentially a lottery where more likely tokens have more « tickets. »

The problem is that after an opening brace ({), the model might assign a 70% probability to a quotation mark ("), but still leave a 2% chance for an unexpected space, or a 0.5% chance of producing something like « Here’s the JSON you requested: » instead. Those probabilities are rarely, if ever, exactly zero.

Even at temperature 0, where the model always selects the most likely token, there’s no guarantee that the most likely token is the correct one. During training, the model has seen countless polite preambles before answers, so it may genuinely consider them more likely than starting the JSON object immediately.

The consequence is straightforward: as long as the model is free to generate any token at every step, it will eventually produce an invalid one. The failure modes are remarkably consistent: an unclosed brace or bracket, a trailing comma, a hallucinated key, an incorrect value type (for example, a number instead of a string), or a perfectly valid JSON object wrapped inside a Markdown code block.

Retries and post-generation validation can reduce the failure rate, but they can never eliminate it.

The only robust solution is to prevent the model from generating any token that could violate the expected format in the first place.

From the Simplest to the Strictest Approaches

JSON Mode

This is the most basic level of structured generation. You ask the model to produce syntactically valid JSON. That guarantees the output can be parsed, but not that it follows a specific structure. The model is still free to invent new keys, omit expected ones, or change value types from one response to the next.

Tool Calling / Function Calling

A more constrained approach is to ask the model to call a function whose arguments follow a predefined schema. This provides much stronger guarantees than plain JSON Mode and has become the standard entry point for structured generation in most modern LLM APIs.

Structured Outputs

The strictest approach is to provide a complete JSON Schema describing the expected fields, types, and allowed values. The model’s output is then constrained to match that schema exactly.

Providers such as OpenAI and Groq refer to this feature as Structured Outputs. In the research literature, however, the underlying technique is known as constrained decoding. They’re simply two perspectives on the same mechanism: Structured Outputs is the API feature you enable, while constrained decoding is what actually happens during token generation.

The Underlying Mechanism: Logit Masking

The core idea is surprisingly simple:

At every generation step, we know which tokens keep the output valid. Every other token is forbidden.

This is the mechanism behind constrained decoding.

In practice, before the logits are passed through the softmax function, the scores of all invalid tokens are replaced with negative infinity. Once softmax is applied, those tokens receive a probability of exactly zero—not 0.001%, but zero. They are completely removed from the sampling process.

The model itself isn’t retrained or modified. Instead, its available choices are filtered at the last possible moment, immediately before the next token is selected.

The obvious question then becomes: how do we know which tokens are valid at each step?

For simple formats—a regular expression, a date, or a value chosen from a fixed list—a finite-state automaton (FSA) is sufficient. The automaton keeps track of its current state and allows only the transitions that preserve a valid output.

JSON, however, is fundamentally more complex because it is recursive. If the model opens three nested objects, it must close exactly three objects, in the correct order. A finite-state automaton has no unbounded memory, so it cannot correctly track arbitrarily deep nesting.

This is where context-free grammars (CFGs) come into play. Formats such as GBNF, used by llama.cpp, describe recursive grammar rules that naturally support nested structures of arbitrary depth while guaranteeing that braces, brackets, and other delimiters are always properly balanced.

Libraries such as Outlines go one step further. Instead of writing a grammar manually, you simply provide a schema—typically a Pydantic model or JSON Schema—and the library automatically compiles it into the internal grammar required to constrain generation. The result is the same guarantee, without having to author the grammar yourself.

The Cost—and the Real Debate

The masking itself is inexpensive. Applying a mask to roughly 130,000 logits is a lightweight GPU operation. The real overhead comes earlier, when the schema or grammar is compiled into an internal representation that can be used during decoding. Fortunately, this is typically a one-time initialization cost that can be cached and reused across requests.

The more interesting question is a conceptual one:

Does constraining the model from the very first token hurt its ability to reason?

It’s a legitimate concern. If the model is forced to begin immediately with an opening brace ({), it loses the opportunity to develop an intermediate line of reasoning before producing the final answer.

The solution that has emerged in practice is to separate reasoning from output formatting.

The model is first allowed to reason freely in natural language. Only the final response—the part consumed by the application—is constrained by the schema. This pattern appears frequently in structured outputs: an unconstrained explanation or reasoning field is followed by the structured fields that capture the actual conclusion.

As a result, invalid outputs don’t merely become less likely.

They become impossible by construction—at least with respect to anything the grammar can express. The grammar guarantees the shape of the output, not the correctness of its content.

A Real-World Example: Mission Radar AI

Mission Radar AI is a freelance opportunity matching platform that I’m currently developing. Although it isn’t in production yet, the reliability requirements are already the same.

The platform automatically analyzes job postings published on LinkedIn, and an LLM extracts structured information such as the daily rate, technology stack, contract type, remote work availability, and required skills. That data is then consumed directly by the application.

In this context, an invalid output format isn’t just an inconvenience—it’s an application bug, whether you’re running the system in development or in production.

Until recently, the processing pipeline looked like this:

De la validation Pydantic pour attraper les JSON mal formés, des retries automatiques via Tenacity pour relancer une génération en cas d’échec. Ça marche, mais ça détecte l’erreur après coup plutôt que de l’empêcher. Mon hypothèse, en découvrant les structured outputs : si mes retries existent à cause de JSON invalide, et que le constrained decoding rend le JSON invalide impossible, je peux simplifier cette partie de l’architecture.

An Important Detail: The LLM Doesn’t Extract Everything

Before going any further, there’s an important detail that changes the picture significantly.

Extraction in Mission Radar AI is not entirely delegated to the LLM. Instead, the platform uses a hybrid architecture.

A deterministic Rule Engine handles the vast majority of fields using regular expressions, pattern matching, and traditional business rules. The LLM is only invoked for low-confidence fields—cases where simple rules are not sufficient to reach a reliable conclusion.

This dramatically reduces the system’s risk surface. Even if the LLM fails to extract a particular field correctly, the rest of the object has already been populated deterministically by the Rule Engine.

In other words, the LLM is not a single point of failure. It’s just one component in the pipeline, reserved for the ambiguous cases where deterministic logic reaches its limits.

The Four Levels of Reliability Applied to extract_structured

extract_structured is the method that takes raw text—for example, a LinkedIn job post—and a Pydantic schema, then asks the LLM to produce data that conforms to that schema: the daily rate, technology stack, contract type, remote status, and so on.

Over time, I found it useful to think of this method in terms of four distinct reliability levels, ranging from the weakest guarantees to the strongest.

Level 1 — Prompt Engineering Only (Weak Guarantee, ~70–80%)

At the most basic level, you simply instruct the LLM to « respond in JSON » and include an example of the expected format in the prompt.

This often works—but only most of the time. The model may prepend explanatory text such as « Here’s the JSON: », wrap the output in Markdown code fences, forget to close a brace, or otherwise produce syntactically invalid JSON.

On its own, this approach is never reliable enough for production. It’s acceptable only for non-critical use cases where occasional formatting errors are tolerable.

Level 2 — Provider JSON Mode (Moderate Guarantee)

Like OpenAI, providers such as Groq and Claude expose a parameter similar to response_format={"type": "json_object"}.

With JSON Mode enabled, the provider ensures that the model produces syntactically valid JSON: braces are properly balanced, quotation marks are correctly placed, and the output can always be parsed.

The limitation is that syntax is the only guarantee. JSON Mode does not ensure that the output matches your expected schema. For example, a field defined as stack: list[str] may still be returned as a single string like "Python, FastAPI" instead of an array.

Level 3 — Function Calling / Tool Use (The First Truly Reliable Approach)

Instead of asking the LLM to generate JSON as free-form text, you define a tool whose arguments follow a JSON Schema generated directly from your Pydantic model.

The model no longer writes JSON itself. Instead, it invokes the tool by supplying arguments that conform to the declared schema. This provides much stronger guarantees than plain JSON Mode and significantly reduces formatting errors.

from pydantic import BaseModel

class ExtractedMission(BaseModel):
    tjm: int | None
    stack: list[str]
    remote: bool

# The Pydantic model is converted directly into a JSON Schema
schema = ExtractedMission.model_json_schema()

# Claude or Groq receive this schema as a tool definition (input_schema)
# → instead of generating JSON as plain text,
#   the model populates the arguments of a function call

The LLM is no longer generating a string that’s supposed to look like JSON. Instead, it’s populating the arguments of a function call whose interface is enforced by the API itself.

In practice, the difference comes down to what the model is actually allowed to produce:

Level 4 — Validation + Retries: Belt and Suspenders

Even with function calling, it’s worth having a final safety net.

The response is validated against the Pydantic model, and if a ValidationError is raised, the request is automatically retried while feeding the validation error back to the LLM as additional context.

from tenacity import retry, stop_after_attempt

@retry(stop=stop_after_attempt(3))
async def extract_structured(self, prompt: str, schema: type[BaseModel]) -> dict:
    raw = await self._call_llm_with_tool(prompt, schema)
    return schema.model_validate(raw).model_dump()  # ValidationError → Tenacity retry

Tenacity is a Python library that handles retries. It automatically re-executes a function when it raises an exception according to a configurable policy—maximum number of attempts, delays between retries, exponential backoff, and so on.

In this case, its role is straightforward: whenever Pydantic validation fails, the LLM is called again with the validation error included in the next prompt.

For extract_structured, I ultimately settled on Level 3 (function calling) combined with Level 4 (Pydantic validation + Tenacity retries).

Plain JSON Mode (Level 2) simply isn’t robust enough once your schema includes anything beyond trivial types—typed lists, enums, optional fields, or more complex nested structures.

Testing Before Changing Anything

Changing an existing architecture based solely on a provider announcement—no matter how compelling—isn’t engineering. It’s a gamble.

In my project, I’m using llama-3.3-70b-versatile, which doesn’t support strict mode (constrained decoding) on Groq. My initial hypothesis was straightforward: by migrating to a model that supports strict mode, I should be able to reduce JSON parsing and validation failures.

Before changing anything, however, I wanted to validate that assumption with DeepEval rather than take it on faith.

A direct comparison wasn’t possible because llama-3.3-70b-versatile doesn’t support constrained decoding. Instead, I compared two different configurations:

• llama-3.3-70b-versatile using standard JSON Mode (the current production baseline).
• openai/gpt-oss-20b using strict mode with constrained decoding.

The evaluation consisted of two DeepEval runs:

• A first run on 8 handcrafted job posts.
• A second run on 16 mixed samples, including 8 real-world LinkedIn posts containing the kind of noise you’d expect in production.

The outcome was unexpected.

Zero failures.

Both models successfully completed both evaluation runs without producing a single invalid JSON document or triggering a single validation error.

That doesn’t confirm my original hypothesis—but it doesn’t really disprove it either, at least not at this scale.

The most likely explanation is that Groq’s current models already produce syntactically valid JSON very reliably when using the standard json_object mode for this kind of extraction task.

The real advantages of constrained decoding are more likely to appear with more complex schemas, smaller or less capable models, or much larger evaluation datasets. If the true failure rate is only 1–2%, a benchmark of 16 samples simply isn’t large enough to reveal a statistically meaningful difference.

The Main Takeaway: Two Approaches, Equivalent in This Case

The most important finding from this benchmark isn’t that strict mode won, or that standard JSON Mode did.

It’s that, for this particular use case, they performed identically.

Standard JSON Mode produced zero failures. Constrained decoding also produced zero failures.

Going back to the original question—does strict mode reduce JSON parsing and validation errors?—the answer, at least in this benchmark, is simply that both approaches are equally effective.

At this scale, the results provide no evidence in favor of migrating, but no evidence against it either. They simply suggest that, for this extraction task and these particular models, output formatting is no longer the limiting factor.

A More Interesting Finding: What the Benchmark Reveals About the Models

Once that equivalence is established, a second observation emerges—this time about quality, rather than format.

So far, only one extraction-quality metric has been evaluated beyond the JSON failure rate: title fidelity.

On that metric, llama-3.3-70b-versatile (70 billion parameters) outperformed openai/gpt-oss-20b (20 billion parameters), achieving a score of 0.812 versus 0.750.

The smaller model was more likely to include raw text copied from the LinkedIn post or to truncate the extracted title.

Based on this metric alone, the larger model appears to produce more faithful extractions. That’s not particularly surprising given the difference in model size, but it’s still worth documenting rather than dismissing.

In other words, strict mode doesn’t compensate for the capability gap between the two models. It guarantees the structure of the output, but not necessarily the quality of the information being extracted.

The Decision—for Now

At this point, there’s no compelling reason to migrate to openai/gpt-oss-20b in strict mode.

It doesn’t improve reliability: both approaches achieved zero failures.

It doesn’t improve extraction quality either: on the only quality metric evaluated so far, the current 70B model outperformed the 20B candidate.

That said, the strict-mode implementation remains in the repository. It’s been tested, documented, and is ready to be reused if a more capable model supporting constrained decoding becomes available in the future.

Three Lessons Learned

First, a new feature announced by an LLM provider is not, by itself, a reason to redesign an existing architecture. It’s a hypothesis worth testing—and sometimes the first obstacle is simply practical. In this case, the production model didn’t even support strict mode.

Second, a benchmark where both approaches achieve zero failures is not a meaningless result. It’s evidence of equivalence, and equivalence is valuable information. It tells us that, for this specific use case, the new technique delivers no measurable benefit, but also no measurable drawback.

Finally, constrained decoding remains a technique I’ll keep in my toolbox.

It simply wasn’t the right lever for the problem Mission Radar AI needs to solve today.

---

If you enjoyed this article, you can follow my work on:

• Substack: Stefano Alletti on Substack
• LinkedIn: Stefano Alletti on LinkedIn

I regularly publish deep dives on LLMs, AI engineering, system design, and lessons learned from building real-world AI applications.