Fixing Your Recurring Outage: Ask for a Schema, Not for JSON

If you build applications on top of large language models (LLMs), you have experienced this failure. Your service, running perfectly for weeks, suddenly starts throwing errors. The culprit? The LLM, which you politely asked to respond in JSON, decided to add a friendly preamble: "Sure, here is the JSON you requested..." Your parser, expecting a {, promptly crashed. You deploy a fix to strip the conversational filler. A week later, it fails again, this time because the model generated a trailing comma.

This cycle of asking for a format, parsing the output, and patching the parser for every new edge case is a recurring outage waiting to happen. It's like asking a contractor to "build a house" and hoping they deliver something with four walls and a roof, rather than giving them a detailed blueprint. The blueprint is a schema. By providing the model with a formal schema, we can constrain its output at the point of generation, guaranteeing a syntactically perfect result every time. This isn't just a better prompt; it's a fundamentally more reliable way to build, moving from hopeful requests to deterministic instructions.

A blueprint, not a request

Imagine you need someone to fill out your tax return. You could hand them a blank sheet of paper and say "please write down my income, expenses, and refund amount, roughly in this order" — and hope they don't also scribble a friendly note in the margin, forget a decimal point, or run out of room and stop mid-sentence. Or you could hand them the actual tax form: labeled boxes, one for each number, nothing else allowed. The second approach doesn't just ask for the right answer — it makes it structurally impossible to hand back something the tax office can't process.

That's the fix described here. LLMs (large language models — the AI systems behind chatbots and many automated tools) are usually asked to produce structured data, such as JSON (a common plain-text format that uses { }, labeled fields, and values, which other software can read automatically), just by being politely told "please respond in JSON." Most of the time that works. Occasionally the model adds a chatty aside, forgets a comma, invents an extra field, or gets cut off partway through — and your program, which was expecting a clean, predictable block of text, breaks.

The better approach, sometimes called constrained or schema-based generation, hands the model an actual form — a schema — that spells out exactly which fields must appear, what type of value belongs in each one, and what's allowed. Crucially, this form isn't just a stronger version of the request: the software generating the text is mechanically blocked, word by word, from writing anything that would violate the form. It's the difference between asking someone nicely to only use blue ink and physically handing them a pen that can only write in blue. The result is text that is guaranteed to be valid, not just usually valid — closing off an entire category of bugs that used to show up as mysterious, recurring outages in production.

Flowchart showing how LLM conversational filler causes parser crashes, and how providing a schema prevents this.

How it works

The shift from asking for structure to demanding it is a core tenet of the modern "prompt as engineering" discipline, which favors testable, structural control over simple prompt templates [1]. To understand the impact of this change, we first need to look at the fragile method it replaces.

The Old Way: "Just Give Me JSON"

For years, the standard way to get structured data from an LLM was to include instructions in the prompt itself.

Prompt:
Analyze the following customer support ticket and provide a summary. Respond ONLY with a JSON object containing the keys "issues", "sentiment", and "confidence".
Ticket:
"My dashboard is taking forever to load and I keep getting a 'payment failed' error when I try to upgrade my account. This is really frustrating."

For a while, this works. The model might return a clean JSON object:

{
  "issues": [
    {"topic": "Slow loading", "count": 1},
    {"topic": "Payment errors", "count": 1}
  ],
  "sentiment": "negative",
  "confidence": 0.87
}

Your application code then takes this raw string output and runs it through a JSON parser. The parsed object is then passed to a downstream system, like a database or an analytics dashboard [2].

The problem is that this process is fundamentally brittle. LLMs are trained to be helpful, conversational text predictors. They are not JSON serializers. There is no guarantee the output string will actually be parsable. Common failure modes include:

Conversational Chit-Chat: The model wraps the JSON in explanatory text. ("Sure, here you go! { ... } Hope this helps!")
Syntax Errors: The model produces technically invalid JSON, such as adding a trailing comma before a closing bracket ("count": 1, }), forgetting quotes around keys, or using single quotes instead of double quotes.
Hallucinated Structure: The model invents new keys that your application doesn't expect ("suggested_solution": "reboot server"), or nests the data in an unexpected way.
Incomplete Output: The model hits its maximum output length mid-generation, resulting in a truncated, unparsable string.

Each of these failures requires a different patch. You can write regexes to strip the conversational parts. You can write "forgiving" parsers that try to fix common syntax errors. You can add retry logic that asks the model to try again, hoping for a better result.

This is a losing battle. You are adding complex, brittle post-processing logic to compensate for the unreliability of the generation step. Every patch is a reaction to a past failure, not a prevention of future ones. The core issue remains: you are treating the LLM's output as an untrusted string that might be JSON.

The New Way: Schema-Constrained Generation

Instead of asking for JSON and hoping for the best, we can force the model to generate valid JSON that conforms to a specific schema. This technique is often called constrained decoding or grammar-based generation.

The magic happens during the token generation process itself. At each step, when the model is deciding which token to output next from its vast vocabulary, we intervene. We use a formal grammar—like a JSON Schema, a regular expression, or an XML schema—to filter the list of all possible next tokens down to only those that are valid at that specific point in the output.

Let's walk through an example. Suppose we provide the model with a schema that requires a JSON object with a key named issues.

Start of Generation: The only valid first token for a JSON object is {. The generation process is constrained to pick { and nothing else.
After {: The schema requires a key. The only valid next token is a " to start the key string.
After ": The schema dictates the key name is issues. The generator is forced to output the tokens i, s, s, u, e, s, and then a closing ".
After "issues": The only valid token is a :.
After :: The schema might specify that the value is an array. The generator is forced to output [.

This process continues, token by token. If the model has just generated {"sentiment": "negative", and the schema says the next key must be "confidence", the generator is physically incapable of starting to write "summary". It is guided along the rails of the schema.

The result is that it's impossible for the model to generate syntactically invalid JSON. It cannot add conversational text, forget a comma, or use the wrong quote style. The output is guaranteed to be parsable because it was constructed according to the rules of the grammar from the very first token.

This approach is implemented in a growing number of tools that fall into the "prompt as engineering" camp, such as the Guidance library, which allows for this kind of structural control over generation [1]. By integrating the schema into the generation loop, these tools eliminate the entire category of parsing failures.

Designing a Schema the Model Can Follow

Constrained generation guarantees syntactic validity, but it doesn't guarantee semantic correctness. The model can still fill your schema with nonsense if you design the schema poorly. The goal is to create a structure that is not only valid but also easy for the model to populate correctly.

Let's design a schema for the support ticket analysis task from source [2]. We want to extract issues, their frequency, the overall sentiment, and a confidence score. We can use a format like JSON Schema to define this.

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "title": "Support Ticket Analysis",
  "type": "object",
  "properties": {
    "issues": {
      "type": "array",
      "description": "List of specific problems mentioned by the user.",
      "items": {
        "type": "object",
        "properties": {
          "topic": {
            "type": "string",
            "description": "A 2-3 word summary of the user's issue, e.g., 'Slow loading' or 'Payment errors'."
          },
          "count": {
            "type": "integer",
            "description": "The number of times this issue was mentioned. Set to 1 if mentioned once."
          }
        },
        "required": ["topic", "count"]
      }
    },
    "sentiment": {
      "type": "string",
      "description": "Overall sentiment of the ticket.",
      "enum": ["positive", "neutral", "negative"]
    },
    "confidence": {
      "type": "number",
      "description": "Confidence in the analysis, from 0.0 to 1.0.",
      "minimum": 0.0,
      "maximum": 1.0
    }
  },
  "required": ["issues", "sentiment", "confidence"]
}

This schema is effective for several reasons:

Simple Types: It uses basic, unambiguous types: string, integer, number, array, object. The model understands these concepts well.
Clear Descriptions: Every field has a description. This acts as a mini-prompt, telling the model exactly what is expected. For topic, the description even includes examples ('Slow loading').
Constrained Choices: For sentiment, we use an enum. This restricts the model's output to one of three valid strings, preventing it from generating synonyms like "unhappy" or "frustrated".
Bounded Numbers: For confidence, we specify a minimum and maximum, ensuring the value is a valid probability.

When designing your own schemas, keep descriptions clear and constraints tight. If a field can only have a few possible values, use an enum. If a string should follow a pattern (like a date), provide a pattern with a regular expression. The more guidance you bake into the schema, the more reliable your output will be.

Flowchart comparing LLM output parsing with and without a schema.

What this means in practice

Adopting schema-constrained generation is more than a technical tweak; it changes how we build, deploy, and pay for LLM-powered features.

For developers, the primary benefit is reliability.

No More Parsers: The endless cycle of writing and patching brittle parsing logic is over. If the generation call succeeds, the output is guaranteed to be valid JSON. This simplifies application code immensely.
Fewer Retries, Lower Cost: The old ask-and-hope method often needs several retries before it produces a usable response, and each retry adds cost and delay for the user. Guaranteeing a valid structure on the first try cuts both the average cost per request and the worst-case wait time. Enforcing the grammar does add a small amount of extra checking during generation, but this is almost always cheaper than paying for a failed attempt and trying again.
Systematic Testing: When output is structured and predictable, it becomes easier to test. Tools like promptfoo can run a batch of test inputs and automatically check that the output always matches the required schema [1], much like a test suite checks ordinary software. This kind of engineering-led testing also makes more ambitious, multi-step systems — such as those built with DSPy, a framework for chaining LLM calls together — far more practical to build and maintain [1].

For end-users, the benefit is a product that just works. Features that rely on LLMs to pull structured data out of messy text — summarizing meeting notes into action items, or sorting customer feedback into categories — become faster and more dependable. Users see fewer loading spinners that end in a cryptic error message. The feature feels less like a trick that might not work this time and more like ordinary, dependable software. An application that promises to extract insights from support tickets will consistently deliver a dashboard with topics and sentiment scores, because the pipeline producing that data is no longer at the mercy of whether the model felt like adding a friendly comment before its answer [2].

Where this is heading

While schema-constrained generation solves the problem of getting well-formed output, it doesn't solve everything. The next set of challenges is about what goes inside the well-formed structure.

First, there's the problem of getting the shape right but the content wrong. A model can be forced to produce a schema-compliant JSON object and still fill it with things that aren't true. If the source text never mentions a payment error, the model might still invent {"topic": "Payment errors", "count": 1} just to have something to put in the required field. Or, if it genuinely can't find an answer, it might quietly insert null or an empty string somewhere it doesn't belong. Today's workaround is to design schemas carefully — making fields optional where a real "I don't know" is possible — and to add a separate check downstream that looks at whether the extracted data is actually true, not just whether it's shaped correctly. It's likely we'll see more "dual-check" systems: one automatic check for correct shape at the moment of generation, and a second check — possibly another LLM call — for factual accuracy before the data gets used for anything important.

Second is the challenge of schema evolution, or what happens when the form itself needs to change. If you add a new field to your schema, how do you roll that out without breaking the other parts of your system that haven't been updated to expect it yet? This is an old, familiar problem in software engineering — it's just new to prompting. Expect the field to borrow solutions that already exist for web APIs, such as keeping multiple versions of a schema alive at once and supporting older versions for a transition period.

Finally, there's a lack of standardization. Right now, the way you hand a schema to a model differs across different AI providers, APIs, and open-source tools like Guidance [1] — a schema written for one tool often needs to be rewritten to work with another. As this technique becomes a basic expectation rather than a novelty, it's likely the industry will settle on a small number of standard ways to describe output constraints, making it far easier to write code that works the same way regardless of which model or provider is behind it.

The move from prompting for JSON to constraining with a schema is a critical step in the maturation of AI engineering. It replaces hope with guarantees, turning LLMs from unpredictable artists into reliable components in a larger software system. The recurring outage is over.