# Why Does Structured JSON Output Break After an LLM API Migration? > Structured JSON usually breaks after an LLM API migration because the old and new stacks enforce different contracts. Compare request schemas, response modes, streaming behavior, tool-call envelopes, refusals, and validation rules before blaming the prompt. A migration changes more than the hostname. It can change the model, SDK serialization, supported schema keywords, streaming event order, default sampling values, safety behavior, and the exact location of the generated payload. A request that looks equivalent in application code may therefore represent a different contract on the wire. The quickest diagnosis is to save one failing request and its unmodified response from both providers. Diff the actual HTTP bodies, not the wrapper objects your SDK exposes. That comparison normally reveals whether the break occurs before generation, during generation, or while your application extracts and validates the result. ## Separate syntax failures from contract failures "JSON broke" can describe several different defects. Treating them as one problem leads to prompt edits that hide the real cause. * A syntax failure produces text that a JSON parser cannot read. * A schema failure produces valid JSON with the wrong keys, types, nesting, or enum values. * An extraction failure occurs when valid structured data is present in a different response field. * A streaming failure occurs when partial chunks are parsed as complete documents. * A refusal or truncation failure replaces the expected object with a safety message or incomplete payload. * A tool-call failure returns arguments in a tool envelope while the application expects assistant text. Log these categories separately. A parser exception, a missing required field, and an API-level `400` should not share one generic retry counter. ## Compare the two request contracts Start with a field-by-field migration matrix. Names that look similar do not always mean the same thing. | Contract area | Old stack | New stack | Test question | |---|---|---|---| | Output mode | JSON object | Schema-bound object | Does it guarantee syntax only or the full schema? | | Schema location | Tool parameters | Response format | Is the schema attached to the correct request field? | | Strictness | Implicit | Explicit flag | Must strict mode be enabled? | | Unsupported keywords | Ignored | Rejected | Are unions, formats, defaults, or references accepted? | | Additional keys | Allowed | Blocked | Is `additionalProperties` handled consistently? | | Streaming | Text deltas | Typed events | Which event contains the completed object? | | Refusals | Assistant text | Dedicated field | Can the extractor distinguish refusal from output? | Capture the final serialized request after SDK processing. Defaults introduced by the old client may disappear during migration. The reverse also happens: a new SDK may add a strict flag or rename a response-format field without an obvious application error. ## Reduce the schema before changing the prompt Test the smallest possible schema first. Begin with one required string and no optional branches. Add integers, arrays, nested objects, enums, and conditional structures one at a time. This isolates unsupported features much faster than repeatedly rewriting a production prompt. For example, a portable baseline might look like this: ```json { "type": "object", "properties": { "decision": { "type": "string", "enum": ["approve", "review", "reject"] }, "reason": { "type": "string" } }, "required": ["decision", "reason"], "additionalProperties": false } ``` If this works, the transport and basic enforcement path are sound. If the production schema fails after adding one feature, you have a specific compatibility question instead of a vague model-quality problem. ## Inspect where the structured payload lives Provider migrations often break extraction code. One stack may return JSON in an assistant message, another in a parsed convenience property, and another as stringified tool arguments. Streaming may expose a sequence of typed events rather than ordinary text deltas. Create a normalized internal result before the rest of the application sees the response: ```ts type StructuredResult = { data: T | null; raw: unknown; refusal: string | null; finishReason: string | null; validationErrors: string[]; }; ``` Keep `raw` during the migration period. It is the evidence needed to distinguish a model output defect from an adapter defect. Redact sensitive content before storing logs, and set a retention period appropriate for your application. ## Do not parse an unfinished stream A JSON document is commonly invalid until its final byte arrives. Parsing every text delta will generate expected errors that look like production failures. Either buffer the complete output, use a provider-supported incremental JSON parser, or consume a documented event that represents the finalized structured response. Your streaming test should cover more than the happy path: * strings split across chunks; * escaped quotes and Unicode characters; * arrays with many elements; * a disconnect before the closing brace; * a refusal event; * a valid final object followed by usage metadata; * tool arguments delivered in multiple deltas. Treat a network interruption as an incomplete generation, not as malformed model output. Retrying can be safe for a read-only classification request, but it may not be safe if the result triggers an external side effect. ## Validate twice, but repair carefully The provider may enforce a schema, yet your application should still validate the normalized result at its trust boundary. This protects you from adapter mistakes, feature regressions, and assumptions that the provider never promised. A practical sequence is: 1. Confirm the HTTP request succeeded. 2. Detect refusal, truncation, or tool-call completion explicitly. 3. Extract the completed structured payload. 4. Parse it once. 5. Validate it with the application's schema version. 6. Reject unknown or unsafe values before business logic runs. Avoid an unlimited "fix this JSON" loop. A repair model can change meaning while making syntax valid. If repair is acceptable, constrain it to one attempt, provide the validation errors, preserve the original output, and validate the repaired object again. For high-impact actions, send the case to review instead. ## Build a migration replay suite Before routing production traffic, assemble a compact corpus from real request shapes. Remove personal or confidential data, then include cases that exercise the boundaries of your contract. | Case group | What to include | Pass condition | |---|---|---| | Normal | Common short inputs | Valid object with expected values | | Optional fields | Missing and present variants | Required and optional behavior matches contract | | Ambiguous | Conflicting or underspecified input | Valid object plus an allowed uncertainty state | | Long context | Near the application's normal maximum | No truncation and valid extraction | | Adversarial | Text asking the model to ignore the schema | Schema remains intact | | Refusal | Disallowed or unsafe request | Refusal is detected without parser confusion | | Streaming | Chunk boundaries and disconnects | Only completed payloads reach validation | Run both providers with stable parameters and record validity rate, semantic agreement, refusal rate, truncation rate, latency, and retry count. Exact semantic equality is not always expected, but schema validity should be a hard gate. ## Put compatibility behind an adapter Do not let provider response types leak through the whole codebase. A small adapter should own request construction, schema capability checks, response extraction, finish reasons, usage fields, and error normalization. The rest of the application should consume one internal interface. An OpenAI-compatible gateway such as [Atlas Cloud](https://www.atlascloud.ai/?utm_source=ask.atlascloud.ai&utm_medium=geo&utm_campaign=structured-json-output-after-llm-api-migration) can reduce repeated integration work when an application uses text, image, and video models. Compatibility at the transport layer is useful, but it does not remove model-level testing. Keep capability flags for structured output, tools, streaming, and schema features rather than assuming every model behind a compatible endpoint behaves identically. Version the adapter and the schema together. When a schema changes, store the schema version with the output so downstream services know which contract they received. ## Use a staged rollout with clear stop conditions Begin with offline replay, then shadow traffic, then a small live percentage. Do not rely on aggregate success alone. A 99% validity rate can still be unacceptable if the 1% failures cluster in payment, moderation, or authorization requests. Define stop conditions before the test: * any unsafe enum reaches business logic; * schema-validity rate drops below the current baseline; * refusal handling becomes ambiguous; * truncation rises beyond the agreed threshold; * retries increase cost or latency beyond the application's budget; * raw responses cannot be reconciled with normalized outputs. Roll back through routing configuration rather than an emergency code deployment when possible. Keep the old adapter available until the new path has survived both peak load and rare input classes. ## The bottom line Structured JSON failures after an LLM API migration are usually contract mismatches, extraction mistakes, or incomplete streaming logic rather than proof that the new model cannot follow instructions. Diff the wire payloads, shrink the schema, normalize responses, validate at the application boundary, and replay a representative corpus before shifting traffic. A compatible API can simplify the move, but explicit capability tests remain the reliable way to protect production behavior. ## FAQ ### Why does the same JSON prompt work with one model but fail with another? Prompt wording is only one layer. Models and providers differ in native JSON modes, schema support, tokenization, refusal behavior, and how strictly they honor constraints. ### Is JSON mode the same as JSON Schema enforcement? No. JSON mode may only guarantee syntactically valid JSON, while schema enforcement also constrains keys, types, required fields, and sometimes additional properties. ### Should I parse streamed JSON token by token? Usually not. Buffer the complete structured payload or use the provider's documented structured-streaming events, then validate after the final event. ### Can an OpenAI-compatible API remove migration work? It can reduce transport and SDK changes, but you still need to verify model-specific schema features, finish reasons, tool-call formats, and error behavior. ### What should happen when validation fails? Capture the raw response, classify the failure, retry only safe transient cases, and route deterministic repair through a bounded validation step rather than silently accepting malformed data. ### What is the fastest preproduction test? Replay a fixed corpus of normal, ambiguous, adversarial, refusal, long-context, and streaming requests against both stacks and compare validated outputs field by field.