App Code

Your app's logic lives in an entrypoint file — inference.py (Python) or src/inference.js (Node.js).

Structure

1from inferencesh import BaseApp, BaseAppInput, BaseAppOutput2from pydantic import Field34class AppSetup(BaseAppInput):5    # Define setup parameters (optional)6    model_id: str = Field(default="gpt2", description="Model to load")78class AppInput(BaseAppInput):9    # Define inputs here10    pass1112class AppOutput(BaseAppOutput):13    # Define outputs here14    pass1516class App(BaseApp):17    async def setup(self, config: AppSetup):18        # Runs once when worker starts or config changes19        pass2021    async def run(self, input_data: AppInput, metadata) -> AppOutput:22        # Runs for each request23        pass2425    async def unload(self):26        # Runs on shutdown27        pass

Defining inputs

1class AppInput(BaseAppInput):2    prompt: str = Field(description="What to generate")3    style: str = Field(default="modern", description="Style to use")4    count: int = Field(default=1, description="How many to generate")

Field types

LLM chat inputs (Python)

Model apps that accept chat-style prompts can use SDK types from inferencesh.models instead of defining every field by hand.

New apps (v0.7.9+): subclass ChatInput and put sampling parameters in model_settings:

1from inferencesh.models import ChatInput, ModelSettings, LLMOutput23class AppInput(ChatInput):4    pass56# Callers pass JSON like:7# {"text": "...", "model_settings": {"temperature": 0.7, "max_tokens": 1024}}

ModelSettings fields are all optional. Omitted fields are not sent to the provider, so each model app can define its own defaults.

Existing apps: LLMInput still exposes flat temperature, top_p, and max_tokens on the input object. Migrate to ChatInput when you want a single nested object in the generated schema.

To add only model_settings to a custom input class, mix in ModelSettingsCapabilityMixin from inferencesh.models. Image, file, reasoning, and tool-calling fields use the other capability mixins in inferencesh.models.llm.

Reasoning carry-over (inferencesh ≥ v0.27.0): some providers return opaque state with a response that the next request must carry: a signature over the thinking (Anthropic, Gemini) or encrypted reasoning (OpenAI). Return it on the final LLMOutput as reasoning_state, a map under a key your app owns ({"openrouter": [...]}). The agent stores it on the assistant message and sends it back on that message's context entry on every later turn. Place only your own entry in a provider request. Chat messages from the REST API never include it. build_messages(..., reasoning_state_key=...) (≥ v0.27.1) replays your entry as reasoning_details; reasoning_field=... (≥ v0.27.4) sends each assistant message's own reasoning text back, for providers such as DeepSeek. See Python SDK: App authoring (LLM inputs).

→ Python SDK: App authoring (LLM inputs) · Decision models (contract)

Defining outputs

1class AppOutput(BaseAppOutput):2    result: str = Field(description="Generated text")3    image: File = Field(description="Generated image")

The run method

This is where your logic goes:

1async def run(self, input_data: AppInput, metadata) -> AppOutput:2    # Log progress3    metadata.log("Processing...")45    # Do work6    result = process(input_data.prompt)78    # Return output9    return AppOutput(result=result)

Runtime metadata

The platform passes a metadata object to setup() and run(). Use it to read execution context: which app and worker are running, which GPUs are assigned, and which user and workspace submitted the task.

In Python, type the parameter with Metadata from inferencesh (SDK v0.7.10+ adds task_id, team_id, and user_id; v0.7.13+ adds gpu_ids):

1from inferencesh import Metadata23async def run(self, input_data: AppInput, metadata: Metadata) -> AppOutput:4  namespace = f"{metadata.team_id}:{metadata.task_id}"5  ...

The Python Metadata model also allows extra fields via Pydantic extra="allow" if the runtime forwards additional keys.

Use task_id, team_id, and user_id to namespace shared upstream resources when one API key serves multiple tenants, for example cache keys, object-store prefixes, or per-workspace rate limits.

On Node.js kernels, the same values arrive on context.metadata (snake_case keys from the runtime envelope). Prefer this.context.metadata over a metadata parameter when your app class exposes context.

Setup for models

Load heavy resources in setup. Use setup schemas to define configurable parameters:

1class AppSetup(BaseAppInput):2    model_id: str = Field(default="gpt2", description="Model to load")3    precision: str = Field(default="fp16", description="Model precision")45class App(BaseApp):6    async def setup(self, config: AppSetup):7        # transformers (and torch) must be in requirements.txt8        from transformers import AutoModel9        self.model = AutoModel.from_pretrained(config.model_id)

This runs once per configuration. Each distinct set of setup values runs as its own instance of the app.

Setup parameters are how one app offers several models or modes; they replaced app variants. See Setup Parameters for how they run and what they cannot change.

Multi-function apps

Apps can expose multiple functions, each with their own input/output types.

1from pydantic import BaseModel23class GreetInput(BaseModel):4    name: str = "World"56class GreetOutput(BaseModel):7    message: str89class ReverseInput(BaseModel):10    text: str1112class ReverseOutput(BaseModel):13    reversed_text: str1415class App:16    async def run(self, input_data: GreetInput) -> GreetOutput:17        return GreetOutput(message=f"Hello, {input_data.name}!")1819    async def reverse(self, input_data: ReverseInput) -> ReverseOutput:20        return ReverseOutput(reversed_text=input_data.text[::-1])

Python: Functions are discovered automatically if they have type hints using Pydantic models.

Node.js: Functions are discovered automatically if matching {PascalName}Input and {PascalName}Output Zod schemas are exported.

Each function can also declare the credentials it uses and tool annotations: see Function metadata.

Calling functions

1curl -X POST https://api.inference.sh/apps/run \2  -H "Authorization: Bearer inf_your_key" \3  -d '{"app": "namespace/name", "function": "reverse", "input": {"text": "hello"}}'

Calling a vendor API

inferencesh.upstream (Python, inferencesh 0.29+) is the shared client of API-wrapper apps: auth from the platform's injected secrets, retries on 429 and 5xx, error mapping, and request assembly from an operation descriptor. Apps made by belt app init --openapi use it through _generated.py; a hand-written wrapper can use it directly.

1import logging23from inferencesh import BaseApp, BaseAppInput, BaseAppOutput, OutputMeta4from inferencesh.upstream import Auth, Operation, UpstreamClient, UpstreamError5from pydantic import Field67search = Operation(8    name="search", method="POST", path="/v1/search",9    query_params=("verbose",),            # wire names; the rest of the input is the JSON body10    annotations={"title": "Search", "readOnlyHint": True, "openWorldHint": True},11)12get_item = Operation(name="get_item", method="GET", path="/v1/items/{id}", path_params=("id",), body="none")1314class SearchInput(BaseAppInput):15    query: str16    verbose: bool | None = None1718class SearchOutput(BaseAppOutput):19    results: list[dict] = Field(default_factory=list)20    raw: dict = Field(default_factory=dict)2122class App(BaseApp):23    async def setup(self, config):24        self.logger = logging.getLogger(__name__)25        self.client = UpstreamClient(26            "https://api.acme.com",27            auth=Auth(kind="api_key", location="header", name="X-API-Key", secret="ACME_API_KEY"),28            logger=self.logger,29        )3031    async def search(self, input_data: SearchInput) -> SearchOutput:32        data = await self.client.call(search, input_data)   # raises UpstreamError on a non-2xx answer33        return SearchOutput(results=data.get("results", []), raw=data, output_meta=OutputMeta(inputs=[], outputs=[]))3435    async def unload(self):36        await self.client.aclose()

• Auth mirrors OpenAPI security schemes: api_key (header, query or cookie), http (bearer or basic) and oauth2. With flow="client_credentials", token_url, client_id_secret and client_secret_secret, the client fetches the token and caches it until it expires; other flows send the injected access token as a bearer. Every secret is the name of an environment variable, read per request. A missing one raises MissingSecret, so the app reports a setup problem rather than a vendor error.
• Operation says where each input field goes: path_params, query_params, header_params by wire name (the model's alias when it has one), everything else as the body (json, form, multipart, or none). Only fields the caller set are sent; the vendor applies its own defaults. output_meta never leaves the app.
• UpstreamClient.call returns the decoded body (JSON, else text). call_into(op, input, Model) validates it into a model. 429 and 5xx are retried with Retry-After or exponential backoff, three times by default; UpstreamError carries status and the parsed body.

Several schemes: pass auth={"name": Auth(...), ...} with default_auth, and set Operation(auth="name") on the operations that use another.

Working with files

Input files are downloaded for you:

1image_path = input_data.image.path

Output files are uploaded for you:

1return AppOutput(image=File(path="/tmp/output.png"))

LLM chat inputs (Python)

For text and chat model apps, use ChatInput and ModelSettings from inferencesh.models (SDK v0.7.9+). Sampling parameters live in a nested model_settings object instead of flat fields on the input schema.

1from inferencesh import BaseApp2from inferencesh.models import ChatInput, LLMOutput34class MyInput(ChatInput):5    pass67class App(BaseApp):8    async def run(self, input_data: MyInput, metadata) -> LLMOutput:9        settings = input_data.model_settings10        temperature = settings.temperature if settings else None11        ...

Callers can override generation per request:

1{2  "text": "Summarize this release",3  "model_settings": {4    "temperature": 0.3,5    "max_tokens": 10246  }7}

→ Python SDK: App authoring (LLM inputs)

Decision models (contract)

Apps with category: decision answer typed questions about a piece of state. They return probabilities, not generated text. Every decision app on the Grid — hosted or behind a vendor API — uses the same input and output shape, so callers can swap apps by name without rewriting integration code.

Set category: decision in inf.yml (see App category). In Python, subclass or return the Pydantic types from inferencesh.models (inferencesh ≥ v0.20.0), aligned with the API decision contract.

Input shape

Text fields (state, instructions, option descriptions, score levels) are JSON any: usually a plain string, or structured JSON if your model expects keys. An app whose model only accepts strings can narrow these in its own input type.

Choice question — id keys the answer in the output; instructions tell the model what to decide; options is a list of { "name", "description?" } (name is returned as choice and keys probabilities).

Score question — levels runs low to high; each level's index is its numeric level (0 … n−1).

Noul question — optional criteria.true / criteria.false pin down what yes and no mean.

Vision — DecisionVisionInput extends DecisionInput with images: file URIs every question can see.

Video: DecisionVideoInput extends DecisionVisionInput with videos: file URIs every question can see, after the images. Send state, images, videos or a mix; at least one must be non-empty. The Python helper ships in inferencesh ≥ v0.23.0 (inferencesh.models).

Example task input:

1{2  "state": "Customer asked to cancel after the trial ended yesterday.",3  "choices": [4    {5      "id": "intent",6      "instructions": "What is the customer trying to do?",7      "options": [8        { "name": "cancel", "description": "End the subscription" },9        { "name": "refund", "description": "Get money back" },10        { "name": "support", "description": "Something else" }11      ]12    }13  ],14  "nouls": [15    {16      "id": "churn_risk",17      "instructions": "Is this account likely to churn within 30 days?",18      "criteria": { "true": "Will cancel or downgrade", "false": "Will stay on a paid plan" }19    }20  ]21}

Output shape

Answers are grouped by question kind, each keyed by question id:

Top-level metadata: model (which model answered) and input_tokens (input tokens the model read; decision apps bill on input tokens).

Example fragment:

1{2  "choices": {3    "intent": {4      "choice": "cancel",5      "confidence": 0.91,6      "probabilities": { "cancel": 0.91, "refund": 0.06, "support": 0.03 }7    }8  },9  "scores": {},10  "nouls": {11    "churn_risk": { "noul": 0.78 }12  },13  "model": "myteam/router@latest",14  "input_tokens": 124015}

Authoring (Python)

1from inferencesh import BaseApp2from inferencesh.models import DecisionInput, DecisionOutput, DecisionVisionInput34class App(BaseApp):5    async def run(self, input_data: DecisionInput, metadata) -> DecisionOutput:6        answers = ...  # from your model: keyed by question id7        return DecisionOutput.from_answers(8            answers,9            input_data,10            model="myteam/router",11            input_tokens=tokens_read,12        )

Subclass DecisionVisionInput when the run must pass image file URIs alongside state and questions, and DecisionVideoInput when the model also watches videos (images first, then videos). See the module docstring in inferencesh.models.decision.

→ Python SDK: Decision contract types · App category decision

Next

→ Configuration