OpenAI Compatible

Point any OpenAI client at inference.sh and call every chat model on the platform. The official SDKs, and the tools built on them (Cursor, Continue, LangChain, LlamaIndex, coding-agent harnesses), work unchanged — only the base URL and the model name differ.

1base_url = https://api.inference.sh/openai

Authentication is your API key as a bearer token, the same key as the rest of the API. POST /openai/chat/completions requires scope apps:execute; GET /openai/models requires apps:read.

→ Authentication — creating keys and scopes → Tasks API — the native interface, with setup, sessions, webhooks and files

Quickstart

1from openai import OpenAI23client = OpenAI(4    base_url="https://api.inference.sh/openai",5    api_key="<your inference.sh API key>",6)78completion = client.chat.completions.create(9    model="openrouter/claude-sonnet-5",10    messages=[{"role": "user", "content": "Say hello in five words"}],11)12print(completion.choices[0].message.content)

1curl https://api.inference.sh/openai/chat/completions \2  -H "Authorization: Bearer $INFERENCE_API_KEY" \3  -H "Content-Type: application/json" \4  -d '{5    "model": "openrouter/claude-sonnet-5",6    "messages": [{"role": "user", "content": "Say hello in five words"}]7  }'

/v1/openai is accepted as well, for clients that insist on a /v1 segment.

The model name is an app reference

There is no separate model catalogue. model is the app reference you already use elsewhere on the platform:

1namespace/name            openrouter/claude-sonnet-52namespace/name@version    openrouter/claude-sonnet-5@3gpaneb7

Unversioned references resolve to the app's current version. The response's model field echoes the version that actually ran, so a request for openrouter/claude-sonnet-5 comes back as openrouter/claude-sonnet-5@3gpaneb7 — record it if you need to reproduce a result.

Bare names are not aliased. gpt-5 is not a model here; openai/gpt-6-astra is.

List models

GET /openai/models

Every chat app visible to you, in OpenAI's model-list shape.

1{2  "object": "list",3  "data": [4    {5      "id": "anthropic/claude-opus-5",6      "object": "model",7      "created": 1788554836,8      "owned_by": "anthropic"9    }10  ]11}

The list contains apps whose run function speaks the platform's LLM contract, which the engine determines from the function's declared types at deploy time. Private apps appear only for teams that can see them, so two keys can legitimately get different lists. Image, video and audio apps are not in this list — reach those through the Tasks API.

Chat completions

POST /openai/chat/completions

Supported

Omitted parameters keep the app's own defaults rather than being forced to a gateway default.

Extensions

message.reasoning and delta.reasoning carry a reasoning model's thinking. It is an addition to the spec, so clients that do not know the field ignore it.

Rejected

These return 400 with the offending parameter named, before any work is dispatched or billed:

n (values other than 1), logprobs, top_logprobs, logit_bias, audio, modalities (anything but ["text"]), prediction, web_search_options, moderation, functions combined with tools, and file.file_id content parts — pass file.file_data as a URL or data URI instead.

Ignored

Accepted and discarded, so clients that always send them still work: user, metadata, store, service_tier, parallel_tool_calls, prompt_cache_key, safety_identifier, verbosity.

Streaming

1stream = client.chat.completions.create(2    model="openrouter/claude-sonnet-5",3    messages=[{"role": "user", "content": "Count from one to five"}],4    stream=True,5    stream_options={"include_usage": True},6)7for chunk in stream:8    if chunk.choices:9        print(chunk.choices[0].delta.content or "", end="")

Chunks are chat.completion.chunk objects; the first carries role: "assistant", the last carries finish_reason, and the stream ends with data: [DONE]. Tool calls arrive as indexed fragments whose function.arguments you concatenate per index, exactly as with OpenAI.

Prefer streaming for anything slow. A non-streaming request holds the connection until the task finishes, and an app that has to cold-start a large model can exceed the edge's request ceiling, surfacing as a 524. Streaming sends bytes immediately and is unaffected.

Tool calling

1tools = [{2    "type": "function",3    "function": {4        "name": "get_weather",5        "parameters": {6            "type": "object",7            "properties": {"city": {"type": "string"}},8            "required": ["city"],9        },10    },11}]1213first = client.chat.completions.create(14    model="openrouter/claude-sonnet-5",15    messages=[{"role": "user", "content": "What is the weather in Paris?"}],16    tools=tools,17    tool_choice="required",18)19call = first.choices[0].message.tool_calls[0]2021second = client.chat.completions.create(22    model="openrouter/claude-sonnet-5",23    messages=[24        {"role": "user", "content": "What is the weather in Paris?"},25        first.choices[0].message,26        {"role": "tool", "tool_call_id": call.id, "content": "22C and sunny"},27    ],28)

Tool schemas are forwarded to the provider untouched, so keywords beyond OpenAI's documented subset — enum, format, nested objects — reach the model as written.

Not every model can call tools, and not every one that can supports every tool_choice mode. A model that cannot honour what you asked for fails the request rather than silently answering in prose.

Structured output

1completion = client.chat.completions.create(2    model="openrouter/claude-sonnet-5",3    messages=[{"role": "user", "content": "Give me three primary colors"}],4    response_format={5        "type": "json_schema",6        "json_schema": {7            "name": "colors",8            "strict": True,9            "schema": {10                "type": "object",11                "properties": {"colors": {"type": "array", "items": {"type": "string"}}},12                "required": ["colors"],13                "additionalProperties": False,14            },15        },16    },17)

json_object and json_schema map onto whatever the provider offers — OpenAI's structured outputs, Anthropic's output format, Gemini's response schema. Providers with no equivalent reject the request instead of returning unconstrained text.

Errors

OpenAI's error envelope, so SDK exception handling works as written:

1{2  "error": {3    "message": "unsupported parameter: n (only n=1 is supported)",4    "type": "invalid_request_error",5    "param": "n",6    "code": "unsupported_parameter"7  }8}

On a stream, a failure after the response has begun arrives as an error object in the SSE body, followed by data: [DONE].

What this surface does not cover

The gateway exposes chat completions and nothing else. Use the platform's own API for:

Billing is identical either way — a completion is a task, and it appears in usage and task cost like any other run.