Gateway
LiteLLM
Call Inference.net models from the LiteLLM Python SDK or LiteLLM Proxy, including the US-only endpoint and gateway tags.
LiteLLM is an open-source Python SDK and proxy that puts many LLM providers behind one OpenAI-style interface. Inference.net speaks the OpenAI Chat Completions API, so LiteLLM connects to it through its OpenAI-compatible provider. No custom provider code is needed.
| Setting | Value |
|---|---|
| API base | https://api.inference.net/v1 |
| API base (US-only) | https://us.api.inference.net/v1 |
| API key | Your Inference project API key |
| LiteLLM model | openai/<model>, for example openai/glm-5.2 |
LiteLLM uses the openai/ prefix to pick its OpenAI-compatible adapter. It removes the prefix and sends the rest of the model name to Inference.net. See inference.net/models for model IDs.
Every request is captured by the Inference Gateway with its cost, latency, and full request and response.
Get your API key
Create an Inference project API key in your dashboard under API Keys.
export INFERENCE_API_KEY=<your-project-api-key>Python SDK
Install LiteLLM
pip install litellmSend a request
import os
from litellm import completion
response = completion(
model="openai/glm-5.2",
api_base="https://api.inference.net/v1",
api_key=os.environ["INFERENCE_API_KEY"],
messages=[{"role": "user", "content": "Say hello in one sentence."}],
max_tokens=256,
)
print(response.choices[0].message.content)Stream a response
import os
from litellm import completion
stream = completion(
model="openai/glm-5.2",
api_base="https://api.inference.net/v1",
api_key=os.environ["INFERENCE_API_KEY"],
messages=[{"role": "user", "content": "Explain prompt caching in two sentences."}],
max_tokens=256,
stream=True,
stream_options={"include_usage": True},
)
for chunk in stream:
if chunk.choices and chunk.choices[0].delta.content:
print(chunk.choices[0].delta.content, end="", flush=True)
usage = getattr(chunk, "usage", None)
if usage is not None:
print("\nUsage:", usage.model_dump())To tag a request with gateway headers, pass them in extra_headers:
response = completion(
model="openai/glm-5.2",
api_base="https://api.inference.net/v1",
api_key=os.environ["INFERENCE_API_KEY"],
messages=[{"role": "user", "content": "Summarize this ticket."}],
extra_headers={
"x-inference-task-id": "summarize-ticket",
"x-inference-environment": "production",
},
)LiteLLM Proxy
Add Inference.net models to your config
Add entries to the model_list in your proxy's config.yaml. The example below also shows a US-only alias for the same model.
model_list:
- model_name: glm-5.2
litellm_params:
model: openai/glm-5.2
api_base: https://api.inference.net/v1
api_key: os.environ/INFERENCE_API_KEY
- model_name: glm-5.2-us
litellm_params:
model: openai/glm-5.2
api_base: https://us.api.inference.net/v1
api_key: os.environ/INFERENCE_API_KEY
general_settings:
master_key: os.environ/LITELLM_MASTER_KEYmodel_name is the alias your applications send to the proxy. litellm_params.model is the Inference.net model ID with the openai/ prefix. The proxy reads INFERENCE_API_KEY from its own environment, so set it wherever the proxy runs, including inside Docker.
Recent LiteLLM versions refuse to start the proxy without a strong master_key. Clients then authenticate to the proxy with that key.
Start the proxy
To try a new proxy locally:
pip install "litellm[proxy]"
export LITELLM_MASTER_KEY="sk-$(openssl rand -hex 24)"
litellm --config config.yaml --host 127.0.0.1 --port 4000Restart an existing proxy after you update its config.
Send a request
Call the proxy with the alias from model_name.
curl http://127.0.0.1:4000/v1/chat/completions \
-H "Authorization: Bearer $LITELLM_MASTER_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "glm-5.2",
"messages": [{"role": "user", "content": "Say hello in one sentence."}]
}'View your results
Open the dashboard to see each request, its cost, and its latency. Usage is billed per token to your credit balance.
Tag proxy requests
To set the same tags on every request to a model, add extra_headers to its litellm_params:
- model_name: glm-5.2
litellm_params:
model: openai/glm-5.2
api_base: https://api.inference.net/v1
api_key: os.environ/INFERENCE_API_KEY
extra_headers:
x-inference-environment: productionThe proxy does not forward client request headers by default. To let applications send their own x-inference-task-id or x-inference-environment headers, enable forwarding in general_settings:
general_settings:
master_key: os.environ/LITELLM_MASTER_KEY
forward_client_headers_to_llm_api: trueWith forwarding on, the proxy forwards client headers that start with x-. If a client sends a header that is also set in extra_headers, the extra_headers value is used.
US-only inference
Use https://us.api.inference.net/v1 as the API base to keep requests in US data centers. It serves the same API and the same models. The model ID does not change, so only the base URL is different. You can expose both as separate proxy aliases, as in the glm-5.2-us example above.
Bring your own provider key
To keep paying a provider directly and use the gateway for capture only, send your provider key in x-inference-provider-api-key. For a proxy, set it in extra_headers. LiteLLM resolves os.environ/ values in extra_headers.
- model_name: gpt-4.1-mini
litellm_params:
model: openai/gpt-4.1-mini
api_base: https://api.inference.net/v1
api_key: os.environ/INFERENCE_API_KEY
extra_headers:
x-inference-provider: openai
x-inference-provider-api-key: os.environ/OPENAI_API_KEYFor a provider other than OpenAI, set x-inference-provider-url to its OpenAI-compatible base URL instead of x-inference-provider. See the supported provider URLs.