Check out the newest way to compare different models for a task/agent harness: AutoEvals
Async API

API

Group API

Submit multiple asynchronous inference requests as a single group for easier tracking and webhook notifications.

Learn how to use our Group API to submit multiple inference requests together, perfect for processing related tasks that need to be tracked as a unit. The Group API supports both chat completions and text completions with up to 50 requests per group.

Group API is available for /v1/slow/group/chat/completions, /v1/slow/group/completions, /v1/slow/group/embeddings, and /v1/slow/group/responses endpoints.

You should not mix completion and chat-completion requests in the same group.

Overview

The Group API provides a streamlined way to submit multiple asynchronous inference requests as a single unit. Unlike the Batch API which requires JSONL file uploads, the Group API accepts requests directly in the request body, making it ideal for:

  • Small to medium batches: Process up to 50 requests at once
  • Related tasks: Group related inference requests together
  • Webhook notifications: Get notified when all requests in a group complete
  • Simpler integration: No file uploads or JSONL formatting required
  • Faster implementation: Direct JSON API calls without file management

Group API vs Batch API

FeatureGroup APIBatch API
Maximum requests501,000,000
Input formatJSON array in request bodyJSONL file upload
File managementNot requiredRequired
Use caseSmall batches, quick implementationLarge-scale processing
Webhook supportYesYes
Completion time72 hours24 hours (default), up to 7 days

Getting Started

1. Submit a Group Request

Submit multiple requests together by sending them as an array in the request body:

TypeScript
const response = await fetch(
  "https://api.inference.net/v1/slow/group/chat/completions",
  {
    method: "POST",
    headers: {
      Authorization: `Bearer ${process.env.INFERENCE_API_KEY}`,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({
      requests: [
        {
          model: "google/gemma-3-27b-instruct/bf-16",
          messages: [
            { role: "system", content: "You are a helpful assistant." },
            { role: "user", content: "What is the capital of France?" },
          ],
          max_tokens: 100,
        },
        {
          model: "google/gemma-3-27b-instruct/bf-16",
          messages: [
            { role: "system", content: "You are a helpful assistant." },
            { role: "user", content: "What is the capital of Germany?" },
          ],
          max_tokens: 100,
        },
      ],
      webhook_id: "my-webhook-123", // Optional: attach a webhook for notifications
    }),
  },
);

const result = await response.json();
console.log(result); // { groupId: "group_abc123", groupSize: 2 }

The response will include a group ID and the number of requests:

JSON
{
  "groupId": "group_xY3kL9mN2pQ",
  "groupSize": 2
}

2. Retrieve Group Results

Once your group is processed, retrieve all generation results using the group ID:

TypeScript
const response = await fetch(
  `https://api.inference.net/v1/slow/group/${groupId}/generations`,
  {
    headers: {
      Authorization: `Bearer ${process.env.INFERENCE_API_KEY}`,
    },
  },
);

const result = await response.json();
console.log(result.generations); // Array of all completed generations

The response includes all generations in the group:

JSON
{
  "generations": [
    {
      "id": "gen_abc123",
      "state": "Success",
      "request": {
        "model": "google/gemma-3-27b-instruct/bf-16",
        "messages": [
          {"role": "system", "content": "You are a helpful assistant."},
          {"role": "user", "content": "What is the capital of France?"}
        ],
        "max_tokens": 100
      },
      "response": {
        "id": "gen_abc123",
        "object": "chat.completion",
        "choices": [
          {
            "message": {
              "role": "assistant",
              "content": "The capital of France is Paris."
            },
            "finish_reason": "stop"
          }
        ],
        "usage": {
          "prompt_tokens": 25,
          "completion_tokens": 8,
          "total_tokens": 33
        }
      }
    },
    {
      "id": "gen_def456",
      "state": "Success",
      "request": {
        "model": "google/gemma-3-27b-instruct/bf-16",
        "messages": [
          {"role": "system", "content": "You are a helpful assistant."},
          {"role": "user", "content": "What is the capital of Germany?"}
        ],
        "max_tokens": 100
      },
      "response": {
        "id": "gen_def456",
        "object": "chat.completion",
        "choices": [
          {
            "message": {
              "role": "assistant",
              "content": "The capital of Germany is Berlin."
            },
            "finish_reason": "stop"
          }
        ],
        "usage": {
          "prompt_tokens": 25,
          "completion_tokens": 8,
          "total_tokens": 33
        }
      }
    }
  ]
}

Using Webhooks

Attach a webhook to receive notifications when your group completes processing. Include the webhook_id when submitting the group:

TypeScript
const response = await fetch(
  "https://api.inference.net/v1/slow/group/chat/completions",
  {
    method: "POST",
    headers: {
      Authorization: `Bearer ${process.env.INFERENCE_API_KEY}`,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({
      requests: [
        /* your requests */
      ],
      webhook_id: "my-webhook-123",
    }),
  },
);

When the group finishes, your webhook receives a slow-group.completed notification. Its data.status is completed when all requests in the group finished, or failed when the group could not finish: processing failed, or the group passed its 72-hour completion window. In a failed group, check each entry's state in generations to see which requests finished.

The notification includes:

  • Group ID
  • Completion status
  • Summary of successful and failed requests
  • Custom IDs for each request (if provided)

See our Webhook Documentation for setup instructions.

Text Completions Support

The Group API also supports text completions:

TypeScript
const response = await fetch(
  "https://api.inference.net/v1/slow/group/completions",
  {
    method: "POST",
    headers: {
      Authorization: `Bearer ${process.env.INFERENCE_API_KEY}`,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({
      requests: [
        {
          model: "google/gemma-3-27b-instruct/bf-16",
          prompt: "The capital of France is",
          max_tokens: 10,
        },
        {
          model: "google/gemma-3-27b-instruct/bf-16",
          prompt: "The capital of Germany is",
          max_tokens: 10,
        },
      ],
    }),
  },
);

Embeddings

A group of embeddings requests sends one slow-group.completed webhook that holds every vector of the group.

With "encoding_format": "base64", set "embed_dtype" to choose the value type of each vector:

embed_dtypeBytes per value768-dimension vector in base64
float32 (default)44,096 characters
float1622,048 characters
  • Values are little-endian.
  • float16 halves the webhook body and your storage. Values keep about 3 significant digits.
  • embed_dtype has no effect with "encoding_format": "float".

To decode a float16 vector in Python:

import base64
import numpy as np

vector = np.frombuffer(base64.b64decode(item["embedding"]), dtype="<f2").astype(np.float32)

Image inputs

An image embedding request takes each image as a user message with one image_url part. Send up to 50 images in one request:

{
  "input": [
    [{ "role": "user", "content": [{ "type": "image_url", "image_url": { "url": "https://my-bucket.s3.amazonaws.com/cat.jpg?X-Amz-Expires=259200&..." } }] }],
    [{ "role": "user", "content": [{ "type": "image_url", "image_url": { "url": "data:image/jpeg;base64,/9j/4AAQ..." } }] }]
  ]
}

image_url.url is one of:

  • A base64 data URI (data:image/png, image/jpeg, image/gif or image/webp) of at most 524,288 characters.
  • An https:// URL of at most 4,096 characters, for example a presigned S3 GET URL. A URL is about 2 KB in the request body, where the same image as a data URI is its full size plus a third.

An https:// URL must:

  • Use a public host name: no IP address, localhost or internal name, and no user:password@. The host must resolve to public addresses.
  • Return a PNG, JPEG, GIF or WebP image of at most 384 KiB within 10 seconds. Up to 4 redirects are followed.

A fetch that fails with HTTP 408, 429 or 5xx, a dropped connection or a timeout is tried twice more before the request fails.

S3 serves at least 5,500 GET requests per second per key prefix and answers 503 SlowDown while it scales past that. Above that rate, spread your keys across prefixes (for example, start each key with a short hash).

We fetch the URL when the request runs, not when you submit it. A group can wait up to its 72-hour completion window, so:

  • Presign URLs for at least 72 hours (X-Amz-Expires=259200).
  • A SigV4 presigned URL lasts at most 7 days, and only when you sign it with long-term IAM user credentials. A URL signed with temporary credentials (an IAM role, STS or SSO session) stops working when that session expires, whatever expiry you set.

Errors:

  • A URL that breaks the rules above gets a 400 when you submit it.
  • A URL we cannot fetch fails only its request: that generation's state is Failed and its stateMessage names the input (from 0) and the reason. The other requests in the group still run. Submit the failed request again with a fresh URL.
Could not fetch the image URL of input 3: the server answered HTTP 403 (an expired presigned URL answers 403).

Retry Safely with Idempotency-Key

A group request can time out after the server accepted it. If you retry it without a key, you can create the same group twice. To prevent this, send an Idempotency-Key header, and send the same key and the same body on each retry.

  • Header: Idempotency-Key, 1 to 255 printable ASCII characters.
  • Endpoints: every async create endpoint under /v1/slow/ and /v1/async/, for groups and for single requests.
  • Scope: your team. Keys from other teams never match yours.
  • Lifetime: 1 hour after the first request with the key succeeds.
RequestResponse
First request with a keyThe normal response.
Same key and body, after the first request succeeded200 with the original body (the same groupId) and the header Idempotent-Replayed: true. No new group.
Same key and body, while the first request is running409 with type IdempotencyKeyInProgress. Wait, then retry with the same key.
Same key, different body or endpoint422 with type IdempotencyKeyReused.
Key that is empty, too long, or not printable ASCII400 with type InvalidIdempotencyKey.

Only successful responses are stored. If the first request fails, a retry with the same key runs again. If the first request was interrupted before it finished, retries can get 409 for up to 10 minutes. After that, a retry with the same key runs again.

A key replays the original group for 1 hour. To submit the same requests again as new work, use a new key. After 1 hour, the same key creates a new group, so finish your retries for a group within that hour.

Idempotency is best effort. If the key check is briefly unavailable, the request runs without it, so in rare cases a retry can still create a second group. Make your pipeline tolerate an occasional duplicate group.

Derive the key from what identifies the group, for example a SHA-256 hash of its custom IDs. Then every retry of that group uses the same key.

Python
import hashlib
import json
import os
import time

import requests

def group_key(custom_ids):
    return hashlib.sha256("\n".join(custom_ids).encode()).hexdigest()

def submit_group(group, custom_ids):
    body = json.dumps(group).encode()  # Serialize once; resend the same bytes.
    headers = {
        "Authorization": f"Bearer {os.environ['INFERENCE_API_KEY']}",
        "Content-Type": "application/json",
        "Idempotency-Key": group_key(custom_ids),
    }
    for attempt in range(6):
        try:
            response = requests.post(
                "https://api.inference.net/v1/slow/group/embeddings",
                data=body,
                headers=headers,
                timeout=120,
            )
        except requests.RequestException:
            time.sleep(2**attempt)
            continue
        if response.status_code == 409 or response.status_code >= 500:
            time.sleep(2**attempt)
            continue
        response.raise_for_status()
        return response.json()  # {"groupId": "...", "groupSize": N}
    raise RuntimeError("group submit did not succeed")

Limits and Constraints

  • Maximum requests per group: 50
  • Request format: Direct JSON (no JSONL files required)
  • Supported endpoints:
    • /v1/slow/group/chat/completions
    • /v1/slow/group/completions
    • /v1/slow/group/embeddings
    • /v1/slow/group/responses
  • Completion time: 72 hours
  • Request expiration: Groups expire after 72 hours if not completed

Best Practices

  1. Group related requests: Use groups for requests that logically belong together (e.g., analyzing multiple documents from the same source).

  2. Use webhooks for notifications: Instead of polling, configure webhooks to be notified when your group completes.

  3. Handle individual failures: Some requests in a group may fail while others succeed. Check each generation's status.

  4. Stay under limits: Keep groups to 50 requests or less. For larger batches, use the Batch API.

  5. Include metadata: Add custom IDs or metadata to your requests for easier tracking:

    JSON
    {
      "model": "google/gemma-3-27b-instruct/bf-16",
      "messages": [{"role": "user", "content": "..."}],
      "metadata": {
        "custom_id": "doc_123",
        "type": "summary"
      }
    }

Error Handling

The API validates your request structure immediately. Common errors include:

JSON
{
  "error": {
    "message": "Invalid request body.",
    "type": "BadRequestError",
    "fields": {
      "_errors": ["Unrecognized key(s) in object: 'webhook_url'"]
    }
  }
}

Ensure you use the correct field names:

  • webhook_id (correct)
  • webhook_url (incorrect — this is for the Batch API)

When to Use Group API

Choose the Group API when you need:

  • Quick implementation without file management
  • To process 50 or fewer related requests
  • Webhook notifications for a set of requests
  • Simple JSON-based integration

For larger workloads (50+ requests), consider using the Batch API instead.

On this page