Ask a language model for JSON and most of the time it complies. Every so often it wraps the object in a chatty sentence, truncates mid-string, invents a trailing comma, or decides that unquoted keys are fine. Downstream, your parser disagrees, your retry loop fires, and you pay for a second inference call to fix a problem the first call created. Constrained decoding — also called grammar-based or guided sampling — eliminates that entire class of failure. The serving engine guarantees, at the token level, that the output conforms to a grammar or JSON schema. Not “usually.” Always.
This post walks through how constrained decoding actually works: the logits mask that gets applied at every step, why tokenization makes the problem genuinely hard, what modern serving engines expose through their APIs, and where the technique quietly bites back. If you run inference yourself or consume structured outputs from an API, understanding the mechanism tells you when the guarantee is free and when it costs you latency, quality, or both.
Every Token Is a Sampling Decision
An autoregressive LLM generates one token per step. At each step the model produces a probability distribution over its entire vocabulary — tens of thousands to hundreds of thousands of entries — and a sampler picks the next token from that distribution. Temperature, top-p, and top-k all just reshape this distribution before sampling.
Constrained decoding adds one more reshaping step. Given the current state of the output — the characters generated so far — the constraint engine computes which tokens would keep the output consistent with the target grammar. Tokens that would violate it get their logits forced to negative infinity before sampling runs, which drives their probability to exactly zero. The model physically cannot emit an invalid token. A conceptual sketch:
def sample_step(model, grammar_state, prompt_ids, temperature):
logits = model.forward(prompt_ids)
# Tokens that would keep the output grammar-consistent
allowed = grammar_state.allowed_token_ids()
mask = torch.full_like(logits, float("-inf"))
mask[allowed] = 0.0
logits = logits + mask
probs = torch.softmax(logits / temperature, dim=-1)
next_id = torch.multinomial(probs, 1)
grammar_state.advance(next_id) # move the automaton forward
return next_idTwo properties follow from this design. First, the guarantee is unconditional: 100% of outputs parse, regardless of prompt quality, temperature, or model size. Second, the constraint only shapes the search space — the model still chooses among the allowed continuations using its own learned distribution. When several valid completions exist, the model’s preferences still decide. That is why constrained decoding usually preserves output quality instead of degrading it, and sometimes improves task accuracy simply by removing format failures from the error budget.
The Tokenization Mismatch Problem
The naive description sounds easy: simulate a parser, reject invalid tokens. The hard part is that grammars speak characters while models speak tokens, and the two vocabularies do not line up. A single token can contain half a JSON string, a quote, and the start of a key. Whether a token is valid depends not just on the grammar state but on which characters of it land inside which construct. A token like ", " might be legal at one point in a JSON document and illegal one character later.
Serious implementations compile the target schema into an automaton and then, crucially, precompute the interaction between that automaton and the model’s token vocabulary. The precomputation is where all the engineering effort lives, because doing it per step is prohibitively slow. XGrammar built its reputation on this insight: compile the grammar up front into an efficient token mask cache, so the per-step cost of generating the mask drops to near zero. Schemas are reused across requests in real serving traffic, so the compilation cost amortizes quickly.
The other well-known engines take related approaches. llguidance powers the guidance backend in many serving stacks, and Outlines compiles regular expressions and context-free grammars into finite-state machines. llama.cpp ships GBNF, a grammar format you can feed directly to local models. The differences between them show up in compilation speed, caching behavior, and dialect support — not in the fundamental guarantee, which is the same everywhere.
What Serving Engines Actually Expose
You rarely need to touch logits yourself. vLLM’s structured outputs feature accepts a JSON schema, a regex, a set of choices, or a full context-free grammar through the OpenAI-compatible API. The request parameter is structured_outputs; the older guided_json-style fields were removed in v0.12.0. Under the hood you can pick the XGrammar or llguidance backend, or leave the default auto setting to choose based on the request. One migration note worth knowing: the lm-format-enforcer backend was deprecated starting in v0.33.0, and its regex dialect is Python’s re, while XGrammar and guidance use Rust-style regex. Changing backends can change which tokens are legal and therefore the outputs you get.
The typical client-side pattern is to define a schema in Pydantic and pass its generated JSON schema through. Using the OpenAI Python SDK against a vLLM endpoint:
from enum import Enum
from openai import OpenAI
from pydantic import BaseModel
class CarType(str, Enum):
sedan = "sedan"
suv = "SUV"
truck = "Truck"
class CarDescription(BaseModel):
brand: str
model: str
car_type: CarType
client = OpenAI(base_url="http://localhost:8000/v1", api_key="-")
model = client.models.list().data[0].id
completion = client.chat.completions.create(
model=model,
messages=[
{"role": "user", "content": "Describe an electric hatchback"}
],
extra_body={
"structured_outputs": {
"json": CarDescription.model_json_schema()
}
},
)
car = CarDescription.model_validate_json(
completion.choices[0].message.content
)Hosted APIs converged on the same idea through response_format with a strict JSON schema mode: you supply the schema, the provider runs constrained decoding server-side, and the promise is a syntactically valid instance every time. The details differ — hosted APIs often reject exotic schema keywords rather than fully supporting the spec — but the mechanism is the one described above.
Structural Tags: Constrained Decoding for Agent Formats
Plain “the whole response is this JSON schema” covers a lot of ground, but agent workloads broke the model of it. Modern agent outputs mix free-form reasoning with structured regions: a thinking section, then a tool call wrapped in model-specific markers, then a final answer. Each open model family defines its own tool-calling syntax, and a single response may need several different constraint regimes in sequence.
XGrammar-2, released in May 2026, introduced the Structural Tag: a JSON-based DSL that composes sequences of constrained and free-form regions — a tag with a begin marker, arbitrary text, and an end marker; triggered sections that only activate once a specific token appears; JSON schema regions rendered in different tool-calling styles. The serving engine enforces structure exactly where structure belongs and leaves the model unconstrained where it should be thinking. The project reports up to 80x efficiency gains over the original XGrammar on large structures through cross-grammar caching and batching support, and the engine is integrated into SGLang, vLLM, TensorRT-LLM, and MLC-LLM for strict tool calling.
The takeaway for anyone building agents: if your harness parses tool calls out of raw model output with string splitting and hopes for the best, the same mechanism that guarantees your JSON can guarantee your tool-call format. Most engines now expose this directly rather than requiring every downstream application to reimplement the parsing layer.
Where It Bites Back
Constrained decoding is a syntactic guarantee, and reading it as anything more is the most common mistake. Four failure patterns come up repeatedly:
- Valid structure, wrong content. The grammar forces the model to produce schema-conforming output even when the model has no good answer. A forced
confidence: 0.95is a fabrication with formatting, not a measurement. Keep validating semantics downstream; the grammar only removes parse errors. - Over-constrained schemas degrade quality. Tight enums and deeply nested requirements leave the model few legitimate paths, and forcing it down a narrow corridor mid-reasoning can produce degenerate loops — the same phrase repeated, or fields filled with plausible garbage. Loose schemas plus a final validation pass often outperform maximal strictness.
- Schema compilation is not free. Huge or pathological grammars can take noticeable time to compile into token masks. When schemas are stable across requests, engines cache the compiled form and the steady-state cost is near zero; when every request ships a unique schema — say, one embedding a different long enum list — you pay compilation repeatedly.
- Dialect drift between backends. Regex flavors differ (Rust-style versus Python), and unsupported schema keywords get ignored or rejected depending on the backend and version. Test against the exact engine and backend you deploy, not the one in your notebook.
One more boundary worth respecting: do not constrain reasoning channels. Forcing a model’s chain of thought into a schema tends to hurt the quality of the conclusion — the reasoning needs freedom to wander before it commits. Structural tags exist precisely so the constrained region starts only after the free-form part ends.
Wrapping Up
Constrained decoding turns “the model usually returns valid JSON” into a compile-time-style guarantee: a grammar compiles to token masks, invalid continuations get zero probability, and the parser never fails again. The machinery is mature — vLLM, SGLang, TensorRT-LLM, and every major hosted API expose it through ordinary request parameters. The engineering discipline is knowing what the guarantee does not cover: semantics, reasoning quality, and schema compilation costs are still your problem. Start by converting your most brittle parsing site — the one with the retry loop — to a strict schema, measure the latency difference with your real schemas, and keep the downstream validator. The retries will simply stop firing.