AutoTrainer

Open beta. AutoTrainer is in open beta. It is available to all users. If you have problems, contact us.

Automatic deployment is temporarily disabled. We are opening it up soon. Until then, training still runs end to end, and the trained model is yours to keep: download the weights from the training page and deploy them anywhere, on any provider or your own hardware. You can also deploy it on our platform or contact us for a dedicated deployment sized to your needs. See After training completes.

AutoTrainer is an automatic version of the training loop. You point an existing chat completions call at AutoTrainer and select a teacher model. The teacher model is a large model that already does the task well. The teacher model serves your requests, and each successful response becomes a training sample. When there are sufficient samples, the platform trains a distilled model, deploys it, and moves your traffic to it. Your endpoint and your code do not change. Your cost and latency decrease. Use AutoTrainer for high-volume, single-purpose tasks that have a stable prompt, for example extraction, classification, summarization, and tagging. If you want control of datasets, rubrics, and recipes, use a manual training run. Not ready to train your own model? AutoEvals compare your task’s live traffic against a set of catalog models and recommend the best off-the-shelf option. Start there to find the right model for your task, then come back to AutoTrainer when you want better cost and latency than any off-the-shelf model gives you.

How it works

AutoTrainer is tied to one task. Each AutoTrainer request must identify its task. Samples collect for each task separately. Each task trains its own distilled model and serves a maximum of one distilled model at a time. If you have two different workloads, give each workload its own task. Each task moves through three stages:

Collecting

The teacher model serves your requests. The platform records each successful teacher response as a training sample. Each sample counts toward your minSamples target.

Training

When the task has minSamples successful samples, and autoTrain is on, the platform builds training and eval datasets from the samples. It then trains a distilled model with a pre-configured recipe. The teacher model continues to serve your requests during training.

Live

When autoDeploy is on, the platform deploys the trained model to a dedicated GPU and does a smoke test. Then it moves your traffic to the trained model automatically. You do not change your code. autoDeploy is temporarily disabled; until it opens up, deploy the trained model yourself as described in After training completes.

AutoTrainer does not stop your service. If training, deployment, or routing fails, your requests go to the teacher model and continue to operate.

Quickstart

The fastest way to use AutoTrainer is createAutoTrainClient from @inference/sdk. This function binds a task, a teacher model, a system prompt, and an optional output schema into one typed client.

Install the SDK

npm install @inference/sdk openai zod

Create an AutoTrainer client

import OpenAI from "openai";
import { z } from "zod";
import { createInferenceClient } from "@inference/sdk";

const client = createInferenceClient({
  openai: OpenAI,
  apiKey: process.env.INFERENCE_API_KEY,
});

const personExtractor = client.createAutoTrainClient({
  task: "person-extraction",
  teacherModel: "glm-5.2",
  systemPrompt: "Extract information about people from text.",
  schema: z.object({
    name: z.string(),
    age: z.number(),
  }),
  config: {
    minSamples: 100,
    autoTrain: true,
    autoDeploy: true,
  },
});

Call it like any other model

// Typed as { name: string; age: number }
const person = await personExtractor.run({
  input: "Hello, my name is John and I am 30 years old.",
});

The teacher model serves the first minSamples successful calls. Then the platform trains and deploys the distilled model. After that, the same call returns responses from the distilled model.

For free-form text tasks, omit schema. Then run() resolves to the raw response string. run() also accepts these optional fields together with input:

Field	Description
`metadata`	Key-value tags for the request. The SDK sends them as `x-inference-metadata-*` headers for filters and metrics.
`environment`	Environment label for the request, for example `production` or `staging`.
`projectId`	Replaces the client’s project for this call.

Use the API directly

The SDK is a convenience, not a requirement. An AutoTrainer request is a standard OpenAI-compatible chat completions call plus four x-inference-* headers. You can send it from all languages that have an HTTP client, for example Rust, Go, Java, or Ruby. To make a direct call:

Set the request model to "auto-train". This value is only a marker. The server selects the teacher model or the distilled model.
Send all four x-inference-* configuration headers with each request. The SDK adds these headers for you. Direct calls must include them.

import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "https://api.inference.net/v1",
  apiKey: process.env.INFERENCE_API_KEY,
});

const response = await client.chat.completions.create(
  {
    model: "auto-train",
    messages: [
      { role: "system", content: "Extract information about people from text." },
      { role: "user", content: "John is 30 years old." },
    ],
  },
  {
    headers: {
      "x-inference-task-id": "person-extraction",
      "x-inference-auto-train-teacher-model": "glm-5.2",
      "x-inference-config-min-samples": "100",
      "x-inference-config-auto-train": "true",
      "x-inference-config-auto-deploy": "true",
    },
  },
);

console.log(response.choices[0].message.content);

import os
from openai import OpenAI

client = OpenAI(
    base_url="https://api.inference.net/v1",
    api_key=os.environ["INFERENCE_API_KEY"],
)

response = client.chat.completions.create(
    model="auto-train",
    messages=[
        {"role": "system", "content": "Extract information about people from text."},
        {"role": "user", "content": "John is 30 years old."},
    ],
    extra_headers={
        "x-inference-task-id": "person-extraction",
        "x-inference-auto-train-teacher-model": "glm-5.2",
        "x-inference-config-min-samples": "100",
        "x-inference-config-auto-train": "true",
        "x-inference-config-auto-deploy": "true",
    },
)

print(response.choices[0].message.content)

curl https://api.inference.net/v1/chat/completions \
  -H "Authorization: Bearer $INFERENCE_API_KEY" \
  -H "Content-Type: application/json" \
  -H "x-inference-task-id: person-extraction" \
  -H "x-inference-auto-train-teacher-model: glm-5.2" \
  -H "x-inference-config-min-samples: 100" \
  -H "x-inference-config-auto-train: true" \
  -H "x-inference-config-auto-deploy: true" \
  -d '{
    "model": "auto-train",
    "messages": [
      {"role": "system", "content": "Extract information about people from text."},
      {"role": "user", "content": "John is 30 years old."}
    ]
  }'

use reqwest::Client;
use serde_json::{json, Value};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let response: Value = Client::new()
        .post("https://api.inference.net/v1/chat/completions")
        .bearer_auth(std::env::var("INFERENCE_API_KEY")?)
        .header("x-inference-task-id", "person-extraction")
        .header("x-inference-auto-train-teacher-model", "glm-5.2")
        .header("x-inference-config-min-samples", "100")
        .header("x-inference-config-auto-train", "true")
        .header("x-inference-config-auto-deploy", "true")
        .json(&json!({
            "model": "auto-train",
            "messages": [
                {"role": "system", "content": "Extract information about people from text."},
                {"role": "user", "content": "John is 30 years old."}
            ]
        }))
        .send()
        .await?
        .json()
        .await?;

    println!("{}", response["choices"][0]["message"]["content"]);
    Ok(())
}

package main

import (
	"bytes"
	"encoding/json"
	"fmt"
	"net/http"
	"os"
)

func main() {
	body, _ := json.Marshal(map[string]any{
		"model": "auto-train",
		"messages": []map[string]string{
			{"role": "system", "content": "Extract information about people from text."},
			{"role": "user", "content": "John is 30 years old."},
		},
	})

	req, _ := http.NewRequest(
		"POST",
		"https://api.inference.net/v1/chat/completions",
		bytes.NewReader(body),
	)
	req.Header.Set("Authorization", "Bearer "+os.Getenv("INFERENCE_API_KEY"))
	req.Header.Set("Content-Type", "application/json")
	req.Header.Set("x-inference-task-id", "person-extraction")
	req.Header.Set("x-inference-auto-train-teacher-model", "glm-5.2")
	req.Header.Set("x-inference-config-min-samples", "100")
	req.Header.Set("x-inference-config-auto-train", "true")
	req.Header.Set("x-inference-config-auto-deploy", "true")

	resp, err := http.DefaultClient.Do(req)
	if err != nil {
		panic(err)
	}
	defer resp.Body.Close()

	var result struct {
		Choices []struct {
			Message struct {
				Content string `json:"content"`
			} `json:"message"`
		} `json:"choices"`
	}
	if err := json.NewDecoder(resp.Body).Decode(&result); err != nil {
		panic(err)
	}
	fmt.Println(result.Choices[0].Message.Content)
}

require "json"
require "net/http"
require "uri"

uri = URI("https://api.inference.net/v1/chat/completions")

request = Net::HTTP::Post.new(uri)
request["Authorization"] = "Bearer #{ENV.fetch("INFERENCE_API_KEY")}"
request["Content-Type"] = "application/json"
request["x-inference-task-id"] = "person-extraction"
request["x-inference-auto-train-teacher-model"] = "glm-5.2"
request["x-inference-config-min-samples"] = "100"
request["x-inference-config-auto-train"] = "true"
request["x-inference-config-auto-deploy"] = "true"
request.body = JSON.generate({
  model: "auto-train",
  messages: [
    { role: "system", content: "Extract information about people from text." },
    { role: "user", content: "John is 30 years old." }
  ]
})

response = Net::HTTP.start(uri.host, uri.port, use_ssl: true) do |http|
  http.request(request)
end

result = JSON.parse(response.body)
puts result.dig("choices", 0, "message", "content")

Configuration reference

SDK option	Header	Default	Description
`task`	`x-inference-task-id`	Required	Stable identifier for the task that AutoTrainer distills. Samples collect for each task separately. Each task serves a maximum of one distilled model.
`teacherModel`	`x-inference-auto-train-teacher-model`	Required	The model that serves requests during sample collection. The distilled model learns from its responses. All models that the gateway can route are applicable.
`config.minSamples`	`x-inference-config-min-samples`	Required	The number of successful teacher responses to collect before training starts. The minimum value is 2.
`config.autoTrain`	`x-inference-config-auto-train`	`true`	Controls if training starts automatically when the task has `minSamples` samples. Set it to `false` to collect samples without training.
`config.autoDeploy`	`x-inference-config-auto-deploy`	`true`	Controls if the platform deploys the trained model and moves the traffic automatically. Set it to `false` to train without a traffic change. The teacher model then continues to serve requests. Temporarily disabled; see After training completes.

Notes on the values:

teacherModel: Select a model that already does the task well. The quality of the distilled model is limited by the quality of the teacher samples. For models on Inference.net, use the model identifier, for example glm-5.2, with your Inference API key. Provider models, for example OpenAI or Anthropic, also operate here. Use the same provider headers as other gateway requests.
minSamples: Set a value that is sufficient for the full range of your real inputs. More diverse samples make a better distilled model. For most tasks, a few hundred samples is a good initial value. Only successful teacher responses count toward the target. Errors and refused requests do not count.
autoTrain / autoDeploy: These are independent gates. With autoTrain: true and autoDeploy: false, the platform trains a model, but the teacher model continues to serve. Use this configuration to examine the trained model before you move the traffic.

You can change minSamples, autoTrain, and autoDeploy at all times. The most recent values on incoming requests apply.

Keep the task identity stable

The combination of the task, the system prompt, the output schema, and the teacher model identifies what AutoTrainer distills. Keep all four stable so that the samples collect toward the same model. If you change the prompt, the schema, or the teacher model, a new identity starts. A task serves a maximum of one distilled model at a time:

The new identity collects samples from zero. The teacher model serves its requests. The previous model continues to serve requests that use the old prompt or schema.
When the replacement model completes training, the platform moves the task’s deployment to the new model. During the change, the teacher model serves the task. The change usually takes some minutes. Then the traffic moves to the new model.
The platform records each retrain as a new version of the same deployment. The version history is the retrain record for the task.
If you use an old prompt or schema again, its old model does not come back. A retired identity routes to the teacher model permanently.

Structured output

Give a Zod schema to createAutoTrainClient as shown in the quickstart. Then run() returns parsed, typed output. The SDK sends an OpenAI structured-outputs response_format, and the schema becomes part of the task identity. When you call the API directly:

For structured output, use response_format with the type json_schema. AutoTrainer requests do not support the json_object type.
Each AutoTrainer request must include a system or developer message.
Requests with a schema and requests without a schema are different identities. Do not mix them in one task.

Monitor progress

AutoTrainer progress shows on the page of the task that you defined. To see it:

Open your project in the dashboard.
Go to Observability, then Tasks.
Select the task that you set in your AutoTrainer requests.
Select the Auto-training tab.

The tab shows the samples collected toward the target, the autoTrain and autoDeploy flags, and links to the training job, the model, and the deployment when the platform creates them.

After training completes

The trained model is yours. You own the weights, and you can deploy them anywhere: on our platform, on another provider, or on your own hardware.

Check the evals. Training runs evals automatically, and the results show on the training job page. To test the model further, run your own evals against it with your own rubrics, as often as you want.
Download the weights. Open the training job page and download the model weights. What you do with them is up to you.
Deploy it. Deploy the model on our platform, or contact us and we will help set up a dedicated deployment sized to your traffic and latency needs.

Billing

There is no separate AutoTrainer fee. You pay for the parts that it controls:

Teacher requests bill the same as other inference requests through the gateway.
The training run bills the same as a manual training run.
The deployment bills the same as a dedicated deployment. Retrains use the task’s current deployment again. They do not add a new deployment.

When the distilled model is live, the cost for each request decreases from the teacher-model price to your dedicated deployment price.

Get Started

Gateway

Eval

Deploy

Platform

Datasets

Train

How it works

Quickstart

Use the API directly

Configuration reference

Keep the task identity stable

Structured output

Monitor progress

After training completes

Billing

Next steps

Tasks

Manage deployments

​How it works

​Quickstart

​Use the API directly

​Configuration reference

​Keep the task identity stable

​Structured output

​Monitor progress

​After training completes

​Billing

​Next steps

Tasks

Manage deployments

How it works

Quickstart

Use the API directly

Configuration reference

Keep the task identity stable

Structured output

Monitor progress

After training completes

Billing

Next steps