Setup Parameters

Setup parameters configure an app's initialization steps. Modifying these parameters triggers a full restart (re-running setup()).

Unlike input parameters which change per request, setup parameters control expensive, static resources:

• Model weights
• Hardware precision (fp16/fp32)
• Database connections
• System prompts or behavior flags

Setup parameters replaced app variants. An app that used to ship a fast and a quality variant, or several quantizations of one model, now exposes that choice as a setup parameter.

Defining Setup Parameters

Define an AppSetup class in inference.py. This schema validates configuration on startup.

1from inferencesh import BaseApp, BaseAppInput2from pydantic import Field34class AppSetup(BaseAppInput):5    model_id: str = Field(default="gpt2", description="HuggingFace model ID")6    precision: str = Field(default="fp16", description="Model precision")7    enable_cache: bool = Field(default=True, description="Enable KV cache")89class App(BaseApp):10    async def setup(self, config: AppSetup):11        # Access parameters via config object12        self.model = load_model(13            config.model_id, 14            precision=config.precision15        )

Usage

Provide setup parameters when starting an app. If parameters match an existing running instance for that app, the request executes immediately. If they differ, the instance re-initializes.

CLI

1belt app run user/app-name --input input.json --setup setup.json

Python SDK

1client.run({2    "app": "my-app",3    "setup": {4        "model_id": "meta-llama/Llama-2-7b",5        "precision": "fp16"6    },7    "input": {8        "prompt": "Hello world"9    }10})

REST API

1curl -X POST https://api.inference.sh/apps/run \2  -H "Authorization: Bearer inf_your_key" \3  -H "Content-Type: application/json" \4  -d '{"app": "user/app-name", "setup": {"model_id": "meta-llama/Llama-2-7b"}, "input": {"prompt": "Hello world"}}'

Web app

The app's page shows its setup fields with the inputs, an enum as a dropdown. A run uses the values set there.

Agents

An app used as an agent tool takes a setup block next to its function. Every call the agent makes through that tool uses it.

One app, several models

Give the app one setup field that names the model, with the allowed values as an enum. A run without setup gets the default.

1from typing import Literal2from inferencesh import BaseApp, BaseAppSetup3from pydantic import Field45MODELS = {6    "small": "org/model-small",7    "base": "org/model-base",8}910class AppSetup(BaseAppSetup):11    model: Literal["small", "base"] = Field(default="base", description="Model size to load")1213class App(BaseApp):14    async def setup(self, config: AppSetup):15        self.model = load_model(MODELS[config.model])

How setup values run

• Each distinct set of setup values runs as its own instance of the app on a worker. Two requests with different values do not share a loaded model.
• The scheduler prefers a worker that is already running the app with the same setup values. A value no worker has loaded starts cold.
• Fields the request leaves out take their defaults from AppSetup.

What setup does not change

• Resources. An app has one resources block in inf.yml, and every setup value runs under it. Size it for the largest configuration. See Configuration.
• Price. Pricing expressions read the task's input and usage, not its setup.

Models that need different amounts of memory, or different prices, are separate apps.