Structured Outputs

​

Introduction

JSON is one of the most widely used formats in the world for applications to exchange data.

Structured Outputs is a feature that ensures the model will always generate responses that adhere to your supplied JSON Schema, so you don’t need to worry about the model omitting a required key, or hallucinating an invalid enum value.

Some benefits of Structured Outputs include:

1. Reliable type-safety: No need to validate or retry incorrectly formatted responses
2. Simpler prompting: No need for strongly worded prompts to achieve consistent formatting

​

Getting Started

You’ll need an Inference Cloud account and API key to use the Batch API. See our Quick Start Guide for instructions on how to create an account and get an API key.

Install the OpenAI SDK for your language of choice. To connect to Inference Cloud using the OpenAI SDK, you will need to set thebaseURL to https://api.inference.net/v1 and the apiKey to your Inference Cloud API key, as shown below:

​

Supported models

See our models table for a list of models that support Structured Outputs.

​

When to use Structured Outputs

Structured Outputs are suitable when you want to indicate a structured schema for use when the model responds to the user.

For example, if you are building a math tutoring application, you might want the assistant to respond to your user using a specific JSON Schema so that you can generate a UI that displays different parts of the model’s output in distinct ways.

Put simply:

• If you are connecting the model to tools, functions, data, etc. in your system, then you should use function calling
• If you want to structure the model’s output when it responds to the user, then you should use a structured response_format

​

Structured Outputs vs JSON mode

Structured Outputs is the evolution of JSON mode. While both ensure valid JSON is produced, only Structured Outputs ensure schema adherance. Both Structured Outputs and JSON mode are supported in the Chat Completions API and Batch API.

We recommend always using Structured Outputs instead of JSON mode when possible.

	Structured Outputs	JSON Mode
Outputs valid JSON	Yes	Yes
Adheres to schema	Yes (see supported schemas)	No
Enabling	response_format: { type: "json_schema", json_schema: {"strict": true, "schema": ...} }	response_format: { type: "json_object" }

​

Example

​

Chain of thought

You can ask the model to output an answer in a structured, step-by-step way, to guide the user through the solution.

​

Example response

​

Defining Structured Outputs with OpenAI SDK

In addition to supporting JSON Schema in the REST API, the OpenAI SDK for JavaScript makes it easy to define object schemas using Zod. Below, you can see how to extract information from unstructured text that conforms to a schema defined in code. The Python SDK helper is currently in beta and does not currently support complex schemas.

​

Step By Step Example

You can use Structured Outputs with the new OpenAI SDK helper to parse the model’s output into your desired format, or you can specify the JSON schema directly.

​

Step 1: Define your object

First you must define an object or data structure to represent the JSON Schema that the model should be constrained to follow. See the examples at the top of this guide for reference.

While Structured Outputs supports much of JSON Schema, some features are unavailable either for performance or technical reasons. See here for more details.

To maximize the quality of model generations, we recommend the following:

• Name keys clearly and intuitively
• Create clear titles and descriptions for important keys in your structure
• Create and use evals to determine the structure that works best for your use case

For example, you can define an object like this:

​

Step 2: Supply your object in the API call

You can use the parse method to automatically parse the JSON response into the object you defined.

Under the hood, the SDK takes care of supplying the JSON schema corresponding to your data structure, and then parsing the response as an object.

​

Step 3: Handle edge cases

In some cases, the model might not generate a valid response that matches the provided JSON schema.

This can happen if for example you reach a max tokens limit and the response is incomplete.

​

Manual Schemas

While Structured Outputs supports much of JSON Schema, some features are unavailable either for performance or technical reasons. See here for more details.

To use a manually defined schema, simply specify:

For example:

​

Streaming

You can use streaming to process model responses or function call arguments as they are being generated, and parse them as structured data.

That way, you don’t have to wait for the entire response to complete before handling it. This is particularly useful if you would like to display JSON fields one by one, or handle function call arguments as soon as they are available.

We recommend relying on the SDKs to handle streaming with Structured Outputs. You can find an example of how to stream function call arguments without the SDK stream helper in the function calling guide.

Here is how you can stream a model response with the stream helper:

You can also use the stream helper to parse function call arguments:

​

Supported schemas

Structured Outputs supports a subset of the JSON Schema language.

​

Supported types

The following types are supported for Structured Outputs:

• String
• Number
• Boolean
• Integer
• Object
• Array
• Enum
• anyOf

​

Root objects must not be anyOf

Note that the root level object of a schema must be an object, and not use anyOf. A pattern that appears in Zod (as one example) is using a discriminated union, which produces an anyOf at the top level. So code such as the following won’t work:

​

All fields must be required

To use Structured Outputs, all fields or function parameters must be specified as required.

Although all fields must be required (and the model will return a value for each parameter), it is possible to emulate an optional parameter by using a union type with null.

​

General Schema Limitations

Limitations on the number of properties, enum values, and total string size may vary depending on the model you are using.

​

additionalProperties: false must always be set in objects

additionalProperties controls whether it is allowable for an object to contain additional keys / values that were not defined in the JSON Schema.

Structured Outputs only supports generating specified keys / values, so we require developers to set additionalProperties: false to opt into Structured Outputs.

​

Key ordering

When using Structured Outputs, outputs will be produced in the same order as the ordering of keys in the schema.

​

Some type-specific keywords are not yet supported

Notable keywords not supported include:

• For strings: minLength, maxLength, pattern, format
• For numbers: minimum, maximum, multipleOf
• For objects: patternProperties, unevaluatedProperties, propertyNames, minProperties, maxProperties
• For arrays: unevaluatedItems, contains, minContains, maxContains, minItems, maxItems, uniqueItems

If you turn on Structured Outputs by supplying strict: true and call the API with an unsupported JSON Schema, you will receive an error.

​

For anyOf, the nested schemas must each be a valid JSON Schema per this subset

Here’s an example supported anyOf schema:

​

Definitions are supported

You can use definitions to define subschemas which are referenced throughout your schema. The following is a simple example.

​

Recursive schemas are supported

Sample recursive schema using # to indicate root recursion.

Sample recursive schema using explicit recursion:

​

JSON mode

JSON mode is a more basic version of the Structured Outputs feature. While JSON mode ensures that model output is valid JSON, Structured Outputs reliably matches the model’s output to the schema you specify. We recommend you use Structured Outputs if it is supported for your use case.

When JSON mode is turned on, the model’s output is ensured to be valid JSON, except for in some edge cases that you should detect and handle appropriately.

To turn on JSON mode with the Chat Completions or Assistants API you can set the response_format to { "type": "json_object" }. If you are using function calling, JSON mode is always turned on.

Important notes:

• When using JSON mode, you must always instruct the model to produce JSON via some message in the conversation, for example via your system message. If you don’t include an explicit instruction to generate JSON, the model may generate an unending stream of whitespace and the request may run continually until it reaches the token limit. To help ensure you don’t forget, the API will throw an error if the string “JSON” does not appear somewhere in the context.
• JSON mode will not guarantee the output matches any specific schema, only that it is valid and parses without errors. You should use Structured Outputs to ensure it matches your schema, or if that is not possible, you should use a validation library and potentially retries to ensure that the output matches your desired schema.
• Your application must detect and handle the edge cases that can result in the model output not being a complete JSON object (see below)

Handling edge cases