Check out the newest way to compare different models for a task/agent harness: AutoEvals

Gateway

LiteLLM

Call Inference.net models from the LiteLLM Python SDK or LiteLLM Proxy, including the US-only endpoint and gateway tags.

LiteLLM is an open-source Python SDK and proxy that puts many LLM providers behind one OpenAI-style interface. Inference.net speaks the OpenAI Chat Completions API, so LiteLLM connects to it through its OpenAI-compatible provider. No custom provider code is needed.

SettingValue
API basehttps://api.inference.net/v1
API base (US-only)https://us.api.inference.net/v1
API keyYour Inference project API key
LiteLLM modelopenai/<model>, for example openai/glm-5.2

LiteLLM uses the openai/ prefix to pick its OpenAI-compatible adapter. It removes the prefix and sends the rest of the model name to Inference.net. See inference.net/models for model IDs.

Every request is captured by the Inference Gateway with its cost, latency, and full request and response.

Get your API key

Create an Inference project API key in your dashboard under API Keys.

export INFERENCE_API_KEY=<your-project-api-key>

Python SDK

Install LiteLLM

pip install litellm

Send a request

import os
from litellm import completion

response = completion(
    model="openai/glm-5.2",
    api_base="https://api.inference.net/v1",
    api_key=os.environ["INFERENCE_API_KEY"],
    messages=[{"role": "user", "content": "Say hello in one sentence."}],
    max_tokens=256,
)

print(response.choices[0].message.content)

Stream a response

import os
from litellm import completion

stream = completion(
    model="openai/glm-5.2",
    api_base="https://api.inference.net/v1",
    api_key=os.environ["INFERENCE_API_KEY"],
    messages=[{"role": "user", "content": "Explain prompt caching in two sentences."}],
    max_tokens=256,
    stream=True,
    stream_options={"include_usage": True},
)

for chunk in stream:
    if chunk.choices and chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)
    usage = getattr(chunk, "usage", None)
    if usage is not None:
        print("\nUsage:", usage.model_dump())

To tag a request with gateway headers, pass them in extra_headers:

response = completion(
    model="openai/glm-5.2",
    api_base="https://api.inference.net/v1",
    api_key=os.environ["INFERENCE_API_KEY"],
    messages=[{"role": "user", "content": "Summarize this ticket."}],
    extra_headers={
        "x-inference-task-id": "summarize-ticket",
        "x-inference-environment": "production",
    },
)

LiteLLM Proxy

Add Inference.net models to your config

Add entries to the model_list in your proxy's config.yaml. The example below also shows a US-only alias for the same model.

config.yaml
model_list:
  - model_name: glm-5.2
    litellm_params:
      model: openai/glm-5.2
      api_base: https://api.inference.net/v1
      api_key: os.environ/INFERENCE_API_KEY
  - model_name: glm-5.2-us
    litellm_params:
      model: openai/glm-5.2
      api_base: https://us.api.inference.net/v1
      api_key: os.environ/INFERENCE_API_KEY

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY

model_name is the alias your applications send to the proxy. litellm_params.model is the Inference.net model ID with the openai/ prefix. The proxy reads INFERENCE_API_KEY from its own environment, so set it wherever the proxy runs, including inside Docker.

Recent LiteLLM versions refuse to start the proxy without a strong master_key. Clients then authenticate to the proxy with that key.

Start the proxy

To try a new proxy locally:

pip install "litellm[proxy]"
export LITELLM_MASTER_KEY="sk-$(openssl rand -hex 24)"
litellm --config config.yaml --host 127.0.0.1 --port 4000

Restart an existing proxy after you update its config.

Send a request

Call the proxy with the alias from model_name.

cURL
curl http://127.0.0.1:4000/v1/chat/completions \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "glm-5.2",
    "messages": [{"role": "user", "content": "Say hello in one sentence."}]
  }'

View your results

Open the dashboard to see each request, its cost, and its latency. Usage is billed per token to your credit balance.

Tag proxy requests

To set the same tags on every request to a model, add extra_headers to its litellm_params:

config.yaml
- model_name: glm-5.2
  litellm_params:
    model: openai/glm-5.2
    api_base: https://api.inference.net/v1
    api_key: os.environ/INFERENCE_API_KEY
    extra_headers:
      x-inference-environment: production

The proxy does not forward client request headers by default. To let applications send their own x-inference-task-id or x-inference-environment headers, enable forwarding in general_settings:

config.yaml
general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY
  forward_client_headers_to_llm_api: true

With forwarding on, the proxy forwards client headers that start with x-. If a client sends a header that is also set in extra_headers, the extra_headers value is used.

US-only inference

Use https://us.api.inference.net/v1 as the API base to keep requests in US data centers. It serves the same API and the same models. The model ID does not change, so only the base URL is different. You can expose both as separate proxy aliases, as in the glm-5.2-us example above.

Bring your own provider key

To keep paying a provider directly and use the gateway for capture only, send your provider key in x-inference-provider-api-key. For a proxy, set it in extra_headers. LiteLLM resolves os.environ/ values in extra_headers.

config.yaml
- model_name: gpt-4.1-mini
  litellm_params:
    model: openai/gpt-4.1-mini
    api_base: https://api.inference.net/v1
    api_key: os.environ/INFERENCE_API_KEY
    extra_headers:
      x-inference-provider: openai
      x-inference-provider-api-key: os.environ/OPENAI_API_KEY

For a provider other than OpenAI, set x-inference-provider-url to its OpenAI-compatible base URL instead of x-inference-provider. See the supported provider URLs.

Next steps

On this page