Setup parameters configure an app's initialization steps. Modifying these parameters triggers a full restart (re-running setup()).
Unlike input parameters which change per request, setup parameters control expensive, static resources:
- Model weights
- Hardware precision (fp16/fp32)
- Database connections
- System prompts or behavior flags
Defining Setup Parameters
Define an AppSetup class in inference.py. This schema validates configuration on startup.
python
1from inferencesh import BaseApp, BaseAppInput2from pydantic import Field34class AppSetup(BaseAppInput):5 model_id: str = Field(default="gpt2", description="HuggingFace model ID")6 precision: str = Field(default="fp16", description="Model precision")7 enable_cache: bool = Field(default=True, description="Enable KV cache")89class App(BaseApp):10 async def setup(self, config: AppSetup):11 # Access parameters via config object12 self.model = load_model(13 config.model_id, 14 precision=config.precision15 )Usage
Provide setup parameters when starting an app. If parameters match an existing running instance for that app, the request executes immediately. If they differ, the instance re-initializes.
CLI
bash
1infsh app run user/app-name --input input.json --setup setup.jsonPython SDK
python
1client.run({2 "app": "my-app",3 "setup": {4 "model_id": "meta-llama/Llama-2-7b",5 "precision": "fp16"6 },7 "input": {8 "prompt": "Hello world"9 }10})