# App Code
> Your app's logic lives in an entrypoint file — inference.py (Python) or src/inference.js (Node.js).
**URL:** https://inference.sh/docs/extend/app-code
**Last updated:** 2026-10-15
---
Your app's logic lives in an entrypoint file — `inference.py` (Python) or `src/inference.js` (Node.js).
---
## Structure
```python Python
from inferencesh import BaseApp, BaseAppInput, BaseAppOutput
from pydantic import Field
class AppSetup(BaseAppInput):
# Define setup parameters (optional)
model_id: str = Field(default="gpt2", description="Model to load")
class AppInput(BaseAppInput):
# Define inputs here
pass
class AppOutput(BaseAppOutput):
# Define outputs here
pass
class App(BaseApp):
async def setup(self, config: AppSetup):
# Runs once when worker starts or config changes
pass
async def run(self, input_data: AppInput, metadata) -> AppOutput:
# Runs for each request
pass
async def unload(self):
# Runs on shutdown
pass
```
```javascript Node.js
import { z } from "zod";
export const AppSetup = z.object({
modelId: z.string().default("gpt2").describe("Model to load"),
});
export const RunInput = z.object({
// Define inputs here
});
export const RunOutput = z.object({
// Define outputs here
});
export class App {
async setup(config) {
// Runs once when worker starts or config changes
}
async run(inputData) {
// Runs for each request
}
async unload() {
// Runs on shutdown
}
}
```
---
## Defining inputs
```python Python
class AppInput(BaseAppInput):
prompt: str = Field(description="What to generate")
style: str = Field(default="modern", description="Style to use")
count: int = Field(default=1, description="How many to generate")
```
```javascript Node.js
export const RunInput = z.object({
prompt: z.string().describe("What to generate"),
style: z.string().default("modern").describe("Style to use"),
count: z.number().default(1).describe("How many to generate"),
});
```
### Field types
| Type | Python | Node.js (Zod) |
|------|--------|---------------|
| Text | `str` | `z.string()` |
| Number | `int` / `float` | `z.number()` |
| Boolean | `bool` | `z.boolean()` |
| File | `File` | `z.object({ path: z.string() })` |
| Optional | `Optional[T]` | `z.string().optional()` |
| Array | `List[T]` | `z.array(z.string())` |
| Enum | `Literal["a", "b"]` | `z.enum(["a", "b"])` |
### LLM chat inputs (Python)
Model apps that accept chat-style prompts can use SDK types from `inferencesh.models` instead of defining every field by hand.
**New apps (v0.7.9+):** subclass `ChatInput` and put sampling parameters in `model_settings`:
```python
from inferencesh.models import ChatInput, ModelSettings, LLMOutput
class AppInput(ChatInput):
pass
# Callers pass JSON like:
# {"text": "...", "model_settings": {"temperature": 0.7, "max_tokens": 1024}}
```
`ModelSettings` fields are all optional. Omitted fields are not sent to the provider, so each model app can define its own defaults.
**Existing apps:** `LLMInput` still exposes flat `temperature`, `top_p`, and `max_tokens` on the input object. Migrate to `ChatInput` when you want a single nested object in the generated schema.
To add only `model_settings` to a custom input class, mix in `ModelSettingsCapabilityMixin` from `inferencesh.models`. Image, file, reasoning, and tool-calling fields use the other capability mixins in `inferencesh.models.llm`.
**Reasoning carry-over (inferencesh ≥ v0.27.0):** some providers return opaque state with a response that the next request must carry: a signature over the thinking (Anthropic, Gemini) or encrypted reasoning (OpenAI). Return it on the final `LLMOutput` as `reasoning_state`, a map under a key your app owns (`{"openrouter": [...]}`). The agent stores it on the assistant message and sends it back on that message's `context` entry on every later turn. Place only your own entry in a provider request. Chat messages from the REST API never include it. `build_messages(..., reasoning_state_key=...)` (≥ v0.27.1) replays your entry as `reasoning_details`; `reasoning_field=...` (≥ v0.27.4) sends each assistant message's own reasoning text back, for providers such as DeepSeek. See [Python SDK: App authoring (LLM inputs)](/docs/api/sdk-python#app-authoring-llm-inputs).
→ [Python SDK: App authoring (LLM inputs)](/docs/api/sdk-python#app-authoring-llm-inputs) · [Decision models (contract)](#decision-models-contract)
---
## Defining outputs
```python Python
class AppOutput(BaseAppOutput):
result: str = Field(description="Generated text")
image: File = Field(description="Generated image")
```
```javascript Node.js
export const RunOutput = z.object({
result: z.string().describe("Generated text"),
image: z.object({ path: z.string() }).describe("Generated image"),
});
```
---
## The run method
This is where your logic goes:
```python Python
async def run(self, input_data: AppInput, metadata) -> AppOutput:
# Log progress
metadata.log("Processing...")
# Do work
result = process(input_data.prompt)
# Return output
return AppOutput(result=result)
```
```javascript Node.js
async run(inputData) {
// inputData is validated against RunInput (defaults filled in)
const result = await process(inputData.prompt);
return { result };
}
```
---
## Runtime metadata
The platform passes a **metadata** object to `setup()` and `run()`. Use it to read execution context: which app and worker are running, which GPUs are assigned, and which user and workspace submitted the task.
In Python, type the parameter with `Metadata` from `inferencesh` (SDK **v0.7.10+** adds `task_id`, `team_id`, and `user_id`; **v0.7.13+** adds `gpu_ids`):
```python
from inferencesh import Metadata
async def run(self, input_data: AppInput, metadata: Metadata) -> AppOutput:
namespace = f"{metadata.team_id}:{metadata.task_id}"
...
```
| Field | Description |
|-------|-------------|
| `app_id` | App being executed |
| `app_version_id` | Resolved app version |
| `worker_id` | Worker executing the task |
| `gpu_ids` | GPU device IDs assigned to the worker (for example `["0"]`), when GPUs are allocated |
| `task_id` | Current task ID |
| `team_id` | Workspace that submitted the task |
| `user_id` | User who submitted the task |
The Python `Metadata` model also allows extra fields via Pydantic `extra="allow"` if the runtime forwards additional keys.
Use `task_id`, `team_id`, and `user_id` to namespace shared upstream resources when one API key serves multiple tenants, for example cache keys, object-store prefixes, or per-workspace rate limits.
On Node.js kernels, the same values arrive on `context.metadata` (snake_case keys from the runtime envelope). Prefer `this.context.metadata` over a `metadata` parameter when your app class exposes `context`.
---
## Setup for models
Load heavy resources in `setup`. Use setup schemas to define configurable parameters:
```python Python
class AppSetup(BaseAppInput):
model_id: str = Field(default="gpt2", description="Model to load")
precision: str = Field(default="fp16", description="Model precision")
class App(BaseApp):
async def setup(self, config: AppSetup):
# transformers (and torch) must be in requirements.txt
from transformers import AutoModel
self.model = AutoModel.from_pretrained(config.model_id)
```
```javascript Node.js
export const AppSetup = z.object({
modelId: z.string().default("gpt2").describe("Model to load"),
precision: z.string().default("fp16").describe("Model precision"),
});
export class App {
async setup(config) {
// config is validated against AppSetup — defaults filled in
this.model = await loadModel(config.modelId);
}
}
```
This runs once per configuration. Each distinct set of setup values runs as its own instance of the app.
Setup parameters are how one app offers several models or modes; they replaced app variants. See [Setup Parameters](/docs/apps/setup-parameters) for how they run and what they cannot change.
---
## Multi-function apps
Apps can expose multiple functions, each with their own input/output types.
```python Python
from pydantic import BaseModel
class GreetInput(BaseModel):
name: str = "World"
class GreetOutput(BaseModel):
message: str
class ReverseInput(BaseModel):
text: str
class ReverseOutput(BaseModel):
reversed_text: str
class App:
async def run(self, input_data: GreetInput) -> GreetOutput:
return GreetOutput(message=f"Hello, {input_data.name}!")
async def reverse(self, input_data: ReverseInput) -> ReverseOutput:
return ReverseOutput(reversed_text=input_data.text[::-1])
```
```javascript Node.js
import { z } from "zod";
export const RunInput = z.object({ name: z.string().default("World") });
export const RunOutput = z.object({ message: z.string() });
export const ReverseInput = z.object({ text: z.string() });
export const ReverseOutput = z.object({ reversedText: z.string() });
export class App {
async run(inputData) {
return { message: `Hello, ${inputData.name}!` };
}
async reverse(inputData) {
return { reversedText: inputData.text.split("").reverse().join("") };
}
}
```
**Python:** Functions are discovered automatically if they have type hints using Pydantic models.
**Node.js:** Functions are discovered automatically if matching `{PascalName}Input` and `{PascalName}Output` Zod schemas are exported.
Each function can also declare the credentials it uses and tool annotations: see [Function metadata](/docs/extend/multi-function-apps#function-metadata).
### Calling functions
```bash
curl -X POST https://api.inference.sh/apps/run \
-H "Authorization: Bearer inf_your_key" \
-d '{"app": "namespace/name", "function": "reverse", "input": {"text": "hello"}}'
```
---
## Calling a vendor API
`inferencesh.upstream` (Python, inferencesh 0.29+) is the shared client of API-wrapper apps: auth from the platform's injected secrets, retries on 429 and 5xx, error mapping, and request assembly from an operation descriptor. Apps made by `belt app init --openapi` use it through `_generated.py`; a hand-written wrapper can use it directly.
```python
import logging
from inferencesh import BaseApp, BaseAppInput, BaseAppOutput, OutputMeta
from inferencesh.upstream import Auth, Operation, UpstreamClient, UpstreamError
from pydantic import Field
search = Operation(
name="search", method="POST", path="/v1/search",
query_params=("verbose",), # wire names; the rest of the input is the JSON body
annotations={"title": "Search", "readOnlyHint": True, "openWorldHint": True},
)
get_item = Operation(name="get_item", method="GET", path="/v1/items/{id}", path_params=("id",), body="none")
class SearchInput(BaseAppInput):
query: str
verbose: bool | None = None
class SearchOutput(BaseAppOutput):
results: list[dict] = Field(default_factory=list)
raw: dict = Field(default_factory=dict)
class App(BaseApp):
async def setup(self, config):
self.logger = logging.getLogger(__name__)
self.client = UpstreamClient(
"https://api.acme.com",
auth=Auth(kind="api_key", location="header", name="X-API-Key", secret="ACME_API_KEY"),
logger=self.logger,
)
async def search(self, input_data: SearchInput) -> SearchOutput:
data = await self.client.call(search, input_data) # raises UpstreamError on a non-2xx answer
return SearchOutput(results=data.get("results", []), raw=data, output_meta=OutputMeta(inputs=[], outputs=[]))
async def unload(self):
await self.client.aclose()
```
- **Auth** mirrors OpenAPI security schemes: `api_key` (header, query or cookie), `http` (`bearer` or `basic`) and `oauth2`. With `flow="client_credentials"`, `token_url`, `client_id_secret` and `client_secret_secret`, the client fetches the token and caches it until it expires; other flows send the injected access token as a bearer. Every secret is the name of an environment variable, read per request. A missing one raises `MissingSecret`, so the app reports a setup problem rather than a vendor error.
- **Operation** says where each input field goes: `path_params`, `query_params`, `header_params` by wire name (the model's alias when it has one), everything else as the body (`json`, `form`, `multipart`, or `none`). Only fields the caller set are sent; the vendor applies its own defaults. `output_meta` never leaves the app.
- **UpstreamClient.call** returns the decoded body (JSON, else text). `call_into(op, input, Model)` validates it into a model. 429 and 5xx are retried with `Retry-After` or exponential backoff, three times by default; `UpstreamError` carries `status` and the parsed `body`.
Several schemes: pass `auth={"name": Auth(...), ...}` with `default_auth`, and set `Operation(auth="name")` on the operations that use another.
---
## Working with files
**Input files** are downloaded for you:
```python Python
image_path = input_data.image.path
```
```javascript Node.js
const imagePath = inputData.image.path;
```
**Output files** are uploaded for you:
```python Python
return AppOutput(image=File(path="/tmp/output.png"))
```
```javascript Node.js
return { image: { path: "/tmp/output.png" } };
```
---
## LLM chat inputs (Python)
For text and chat model apps, use `ChatInput` and `ModelSettings` from `inferencesh.models` (SDK **v0.7.9+**). Sampling parameters live in a nested `model_settings` object instead of flat fields on the input schema.
```python
from inferencesh import BaseApp
from inferencesh.models import ChatInput, LLMOutput
class MyInput(ChatInput):
pass
class App(BaseApp):
async def run(self, input_data: MyInput, metadata) -> LLMOutput:
settings = input_data.model_settings
temperature = settings.temperature if settings else None
...
```
Callers can override generation per request:
```json
{
"text": "Summarize this release",
"model_settings": {
"temperature": 0.3,
"max_tokens": 1024
}
}
```
| Type | When to use |
|------|-------------|
| `ChatInput` | New chat/LLM apps — nested `model_settings`, `context`, `system_prompt` |
| `LLMInput` / `BaseLLMInput` | Legacy apps with flat `temperature`, `top_p`, `max_tokens` on the input |
| `ModelSettingsCapabilityMixin` | Custom inputs that need `model_settings` without full `ChatInput` |
→ [Python SDK: App authoring (LLM inputs)](/docs/api/sdk-python#app-authoring-llm-inputs)
---
## Decision models (contract)
Apps with `category: decision` answer **typed questions** about a piece of state. They return **probabilities**, not generated text. Every decision app on the Grid — hosted or behind a vendor API — uses the same input and output shape, so callers can swap apps by name without rewriting integration code.
Set `category: decision` in `inf.yml` (see [App category](/docs/extend/configuration#categories)). In Python, subclass or return the Pydantic types from `inferencesh.models` (**`inferencesh` ≥ v0.20.0**), aligned with the API decision contract.
### Input shape
| Field | Role |
|-------|------|
| `state` | What the questions are about — a string or JSON object the model reads |
| `choices` | Optional list of **choice** questions (pick one of several options) |
| `scores` | Optional list of **score** questions (position on ordered levels) |
| `nouls` | Optional list of **noul** questions (probability that something is true) |
Text fields (`state`, `instructions`, option descriptions, score `levels`) are JSON `any`: usually a plain string, or structured JSON if your model expects keys. An app whose model only accepts strings can narrow these in its own input type.
**Choice question** — `id` keys the answer in the output; `instructions` tell the model what to decide; `options` is a list of `{ "name", "description?" }` (`name` is returned as `choice` and keys `probabilities`).
**Score question** — `levels` runs low to high; each level's index is its numeric level (0 … n−1).
**Noul question** — optional `criteria.true` / `criteria.false` pin down what yes and no mean.
**Vision** — `DecisionVisionInput` extends `DecisionInput` with `images`: file URIs every question can see.
**Video**: `DecisionVideoInput` extends `DecisionVisionInput` with `videos`: file URIs every question can see, after the images. Send `state`, `images`, `videos` or a mix; at least one must be non-empty. The Python helper ships in **`inferencesh` ≥ v0.23.0** (`inferencesh.models`).
Example task input:
```json
{
"state": "Customer asked to cancel after the trial ended yesterday.",
"choices": [
{
"id": "intent",
"instructions": "What is the customer trying to do?",
"options": [
{ "name": "cancel", "description": "End the subscription" },
{ "name": "refund", "description": "Get money back" },
{ "name": "support", "description": "Something else" }
]
}
],
"nouls": [
{
"id": "churn_risk",
"instructions": "Is this account likely to churn within 30 days?",
"criteria": { "true": "Will cancel or downgrade", "false": "Will stay on a paid plan" }
}
]
}
```
### Output shape
Answers are grouped by question kind, each keyed by question `id`:
| Block | Fields |
|-------|--------|
| `choices[id]` | `choice` (highest-probability option), `confidence`, `probabilities` (every option name → probability) |
| `scores[id]` | `score` (probability-weighted level), `normalized` (0–1 vs top level), `confidence`, `probabilities` (level index as string → probability), `legend` (index → level description) |
| `nouls[id]` | `noul` — probability the answer is yes |
Top-level metadata: `model` (which model answered) and `input_tokens` (input tokens the model read; decision apps bill on input tokens).
Example fragment:
```json
{
"choices": {
"intent": {
"choice": "cancel",
"confidence": 0.91,
"probabilities": { "cancel": 0.91, "refund": 0.06, "support": 0.03 }
}
},
"scores": {},
"nouls": {
"churn_risk": { "noul": 0.78 }
},
"model": "myteam/router@latest",
"input_tokens": 1240
}
```
### Authoring (Python)
```python
from inferencesh import BaseApp
from inferencesh.models import DecisionInput, DecisionOutput, DecisionVisionInput
class App(BaseApp):
async def run(self, input_data: DecisionInput, metadata) -> DecisionOutput:
answers = ... # from your model: keyed by question id
return DecisionOutput.from_answers(
answers,
input_data,
model="myteam/router",
input_tokens=tokens_read,
)
```
Subclass `DecisionVisionInput` when the run must pass image file URIs alongside `state` and questions, and `DecisionVideoInput` when the model also watches videos (images first, then videos). See the module docstring in `inferencesh.models.decision`.
→ [Python SDK: Decision contract types](/docs/api/sdk-python#app-authoring-decision-models) · [App category `decision`](/docs/extend/configuration#categories)
---
## Next
→ [Configuration](/docs/extend/configuration)