API
Group API
Submit multiple asynchronous inference requests as a single group for easier tracking and webhook notifications.
Learn how to use our Group API to submit multiple inference requests together, perfect for processing related tasks that need to be tracked as a unit. The Group API supports both chat completions and text completions with up to 50 requests per group.
Group API is available for /v1/slow/group/chat/completions, /v1/slow/group/completions, /v1/slow/group/embeddings, and /v1/slow/group/responses endpoints.
You should not mix completion and chat-completion requests in the same group.
Overview
The Group API provides a streamlined way to submit multiple asynchronous inference requests as a single unit. Unlike the Batch API which requires JSONL file uploads, the Group API accepts requests directly in the request body, making it ideal for:
- Small to medium batches: Process up to 50 requests at once
- Related tasks: Group related inference requests together
- Webhook notifications: Get notified when all requests in a group complete
- Simpler integration: No file uploads or JSONL formatting required
- Faster implementation: Direct JSON API calls without file management
Group API vs Batch API
| Feature | Group API | Batch API |
|---|---|---|
| Maximum requests | 50 | 1,000,000 |
| Input format | JSON array in request body | JSONL file upload |
| File management | Not required | Required |
| Use case | Small batches, quick implementation | Large-scale processing |
| Webhook support | Yes | Yes |
| Completion time | 72 hours | 24 hours (default), up to 7 days |
Getting Started
1. Submit a Group Request
Submit multiple requests together by sending them as an array in the request body:
const response = await fetch(
"https://api.inference.net/v1/slow/group/chat/completions",
{
method: "POST",
headers: {
Authorization: `Bearer ${process.env.INFERENCE_API_KEY}`,
"Content-Type": "application/json",
},
body: JSON.stringify({
requests: [
{
model: "google/gemma-3-27b-instruct/bf-16",
messages: [
{ role: "system", content: "You are a helpful assistant." },
{ role: "user", content: "What is the capital of France?" },
],
max_tokens: 100,
},
{
model: "google/gemma-3-27b-instruct/bf-16",
messages: [
{ role: "system", content: "You are a helpful assistant." },
{ role: "user", content: "What is the capital of Germany?" },
],
max_tokens: 100,
},
],
webhook_id: "my-webhook-123", // Optional: attach a webhook for notifications
}),
},
);
const result = await response.json();
console.log(result); // { groupId: "group_abc123", groupSize: 2 }The response will include a group ID and the number of requests:
{
"groupId": "group_xY3kL9mN2pQ",
"groupSize": 2
}2. Retrieve Group Results
Once your group is processed, retrieve all generation results using the group ID:
const response = await fetch(
`https://api.inference.net/v1/slow/group/${groupId}/generations`,
{
headers: {
Authorization: `Bearer ${process.env.INFERENCE_API_KEY}`,
},
},
);
const result = await response.json();
console.log(result.generations); // Array of all completed generationsThe response includes all generations in the group:
{
"generations": [
{
"id": "gen_abc123",
"state": "Success",
"request": {
"model": "google/gemma-3-27b-instruct/bf-16",
"messages": [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "What is the capital of France?"}
],
"max_tokens": 100
},
"response": {
"id": "gen_abc123",
"object": "chat.completion",
"choices": [
{
"message": {
"role": "assistant",
"content": "The capital of France is Paris."
},
"finish_reason": "stop"
}
],
"usage": {
"prompt_tokens": 25,
"completion_tokens": 8,
"total_tokens": 33
}
}
},
{
"id": "gen_def456",
"state": "Success",
"request": {
"model": "google/gemma-3-27b-instruct/bf-16",
"messages": [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "What is the capital of Germany?"}
],
"max_tokens": 100
},
"response": {
"id": "gen_def456",
"object": "chat.completion",
"choices": [
{
"message": {
"role": "assistant",
"content": "The capital of Germany is Berlin."
},
"finish_reason": "stop"
}
],
"usage": {
"prompt_tokens": 25,
"completion_tokens": 8,
"total_tokens": 33
}
}
}
]
}Using Webhooks
Attach a webhook to receive notifications when your group completes processing. Include the webhook_id when submitting the group:
const response = await fetch(
"https://api.inference.net/v1/slow/group/chat/completions",
{
method: "POST",
headers: {
Authorization: `Bearer ${process.env.INFERENCE_API_KEY}`,
"Content-Type": "application/json",
},
body: JSON.stringify({
requests: [
/* your requests */
],
webhook_id: "my-webhook-123",
}),
},
);When the group finishes, your webhook receives a slow-group.completed notification. Its data.status is completed when all requests in the group finished, or failed when the group could not finish: processing failed, or the group passed its 72-hour completion window. In a failed group, check each entry's state in generations to see which requests finished.
The notification includes:
- Group ID
- Completion status
- Summary of successful and failed requests
- Custom IDs for each request (if provided)
See our Webhook Documentation for setup instructions.
Text Completions Support
The Group API also supports text completions:
const response = await fetch(
"https://api.inference.net/v1/slow/group/completions",
{
method: "POST",
headers: {
Authorization: `Bearer ${process.env.INFERENCE_API_KEY}`,
"Content-Type": "application/json",
},
body: JSON.stringify({
requests: [
{
model: "google/gemma-3-27b-instruct/bf-16",
prompt: "The capital of France is",
max_tokens: 10,
},
{
model: "google/gemma-3-27b-instruct/bf-16",
prompt: "The capital of Germany is",
max_tokens: 10,
},
],
}),
},
);Embeddings
A group of embeddings requests sends one slow-group.completed webhook that
holds every vector of the group.
With "encoding_format": "base64", set "embed_dtype" to choose the value
type of each vector:
embed_dtype | Bytes per value | 768-dimension vector in base64 |
|---|---|---|
float32 (default) | 4 | 4,096 characters |
float16 | 2 | 2,048 characters |
- Values are little-endian.
float16halves the webhook body and your storage. Values keep about 3 significant digits.embed_dtypehas no effect with"encoding_format": "float".
To decode a float16 vector in Python:
import base64
import numpy as np
vector = np.frombuffer(base64.b64decode(item["embedding"]), dtype="<f2").astype(np.float32)Image inputs
An image embedding request takes each image as a user message with one
image_url part. Send up to 50 images in one request:
{
"input": [
[{ "role": "user", "content": [{ "type": "image_url", "image_url": { "url": "https://my-bucket.s3.amazonaws.com/cat.jpg?X-Amz-Expires=259200&..." } }] }],
[{ "role": "user", "content": [{ "type": "image_url", "image_url": { "url": "data:image/jpeg;base64,/9j/4AAQ..." } }] }]
]
}image_url.url is one of:
- A base64 data URI (
data:image/png,image/jpeg,image/giforimage/webp) of at most 524,288 characters. - An
https://URL of at most 4,096 characters, for example a presigned S3 GET URL. A URL is about 2 KB in the request body, where the same image as a data URI is its full size plus a third.
An https:// URL must:
- Use a public host name: no IP address,
localhostor internal name, and nouser:password@. The host must resolve to public addresses. - Return a PNG, JPEG, GIF or WebP image of at most 384 KiB within 10 seconds. Up to 4 redirects are followed.
A fetch that fails with HTTP 408, 429 or 5xx, a dropped connection or a timeout is tried twice more before the request fails.
S3 serves at least 5,500 GET requests per second per key prefix and answers
503 SlowDown while it scales past that. Above that rate, spread your keys
across prefixes (for example, start each key with a short hash).
We fetch the URL when the request runs, not when you submit it. A group can wait up to its 72-hour completion window, so:
- Presign URLs for at least 72 hours (
X-Amz-Expires=259200). - A SigV4 presigned URL lasts at most 7 days, and only when you sign it with long-term IAM user credentials. A URL signed with temporary credentials (an IAM role, STS or SSO session) stops working when that session expires, whatever expiry you set.
Errors:
- A URL that breaks the rules above gets a
400when you submit it. - A URL we cannot fetch fails only its request: that generation's
stateisFailedand itsstateMessagenames the input (from 0) and the reason. The other requests in the group still run. Submit the failed request again with a fresh URL.
Could not fetch the image URL of input 3: the server answered HTTP 403 (an expired presigned URL answers 403).Retry Safely with Idempotency-Key
A group request can time out after the server accepted it. If you retry it
without a key, you can create the same group twice. To prevent this, send an
Idempotency-Key header, and send the same key and the same body on each retry.
- Header:
Idempotency-Key, 1 to 255 printable ASCII characters. - Endpoints: every async create endpoint under
/v1/slow/and/v1/async/, for groups and for single requests. - Scope: your team. Keys from other teams never match yours.
- Lifetime: 1 hour after the first request with the key succeeds.
| Request | Response |
|---|---|
| First request with a key | The normal response. |
| Same key and body, after the first request succeeded | 200 with the original body (the same groupId) and the header Idempotent-Replayed: true. No new group. |
| Same key and body, while the first request is running | 409 with type IdempotencyKeyInProgress. Wait, then retry with the same key. |
| Same key, different body or endpoint | 422 with type IdempotencyKeyReused. |
| Key that is empty, too long, or not printable ASCII | 400 with type InvalidIdempotencyKey. |
Only successful responses are stored. If the first request fails, a retry with
the same key runs again. If the first request was interrupted before it
finished, retries can get 409 for up to 10 minutes. After that, a retry with
the same key runs again.
A key replays the original group for 1 hour. To submit the same requests again as new work, use a new key. After 1 hour, the same key creates a new group, so finish your retries for a group within that hour.
Idempotency is best effort. If the key check is briefly unavailable, the request runs without it, so in rare cases a retry can still create a second group. Make your pipeline tolerate an occasional duplicate group.
Derive the key from what identifies the group, for example a SHA-256 hash of its custom IDs. Then every retry of that group uses the same key.
import hashlib
import json
import os
import time
import requests
def group_key(custom_ids):
return hashlib.sha256("\n".join(custom_ids).encode()).hexdigest()
def submit_group(group, custom_ids):
body = json.dumps(group).encode() # Serialize once; resend the same bytes.
headers = {
"Authorization": f"Bearer {os.environ['INFERENCE_API_KEY']}",
"Content-Type": "application/json",
"Idempotency-Key": group_key(custom_ids),
}
for attempt in range(6):
try:
response = requests.post(
"https://api.inference.net/v1/slow/group/embeddings",
data=body,
headers=headers,
timeout=120,
)
except requests.RequestException:
time.sleep(2**attempt)
continue
if response.status_code == 409 or response.status_code >= 500:
time.sleep(2**attempt)
continue
response.raise_for_status()
return response.json() # {"groupId": "...", "groupSize": N}
raise RuntimeError("group submit did not succeed")Limits and Constraints
- Maximum requests per group: 50
- Request format: Direct JSON (no JSONL files required)
- Supported endpoints:
/v1/slow/group/chat/completions/v1/slow/group/completions/v1/slow/group/embeddings/v1/slow/group/responses
- Completion time: 72 hours
- Request expiration: Groups expire after 72 hours if not completed
Best Practices
-
Group related requests: Use groups for requests that logically belong together (e.g., analyzing multiple documents from the same source).
-
Use webhooks for notifications: Instead of polling, configure webhooks to be notified when your group completes.
-
Handle individual failures: Some requests in a group may fail while others succeed. Check each generation's status.
-
Stay under limits: Keep groups to 50 requests or less. For larger batches, use the Batch API.
-
Include metadata: Add custom IDs or metadata to your requests for easier tracking:
JSON { "model": "google/gemma-3-27b-instruct/bf-16", "messages": [{"role": "user", "content": "..."}], "metadata": { "custom_id": "doc_123", "type": "summary" } }
Error Handling
The API validates your request structure immediately. Common errors include:
{
"error": {
"message": "Invalid request body.",
"type": "BadRequestError",
"fields": {
"_errors": ["Unrecognized key(s) in object: 'webhook_url'"]
}
}
}Ensure you use the correct field names:
webhook_id(correct)webhook_url(incorrect — this is for the Batch API)
When to Use Group API
Choose the Group API when you need:
- Quick implementation without file management
- To process 50 or fewer related requests
- Webhook notifications for a set of requests
- Simple JSON-based integration
For larger workloads (50+ requests), consider using the Batch API instead.