Workers are computers that run your tasks.
Two types
| Type | Description |
|---|---|
| Cloud | Managed by inference.sh |
| Private | Your own hardware |
Cloud workers
- Pay-per-use
- Auto-scaling
- No setup required
Good for getting started and variable workloads.
For programmatic GPU pricing and availability across providers, use the public Instance types API (GET /instances/types).
Private workers
- Run on your hardware
- Via the inference.sh Engine
- Data stays on your network
Good for data privacy, dedicated resources, or cost control.
→ Engines API — drain, update, and lifecycle for private engines
How tasks find workers
- You run an app
- Task goes to the queue
- A worker picks it up when one is available (tasks wait in Queued if capacity is temporarily full)
- Worker runs it and returns results
You can choose cloud vs private when running. Call POST /engines/resources to check which workers match an app's GPU and RAM requirements before you submit a run — see Engines API.
See Tasks for queue timeout and failure behavior.
When no worker is available yet
If every matching worker is busy, offline, or does not meet the app's resource requirements (GPU type, VRAM, CPU), the task stays in Queued instead of failing immediately. The scheduler retries as workers come online.
| Situation | What happens |
|---|---|
| Workers are temporarily busy or restarting | Task waits in the queue; it runs when capacity returns |
App forbids the infra you chose (e.g. cloud-only app with infra: "private") | Task fails right away with a clear error |
| No matching worker exists (wrong GPU, VRAM, or infra) | Task fails after about 10 minutes with no active workers with matching resources |
Matching cloud workers exist but are reserved for other apps (min_concurrency) | Task fails after about 10 minutes with no available workers — capacity reserved for higher-priority apps |
| Workers exist but stay busy or unavailable | Task fails after roughly 24 hours in the queue with task expired while waiting for available capacity, please retry |
Some store apps have a min_concurrency floor — the platform keeps at least that many cloud workers loaded with the app so it stays warm. Those workers cannot be displaced by other apps when doing so would drop below the floor. This only affects cloud scheduling; private workers are not reserved this way. Check min_concurrency on the store listing if you see the reserved-capacity error.
On unified-memory GPUs (for example NVIDIA DGX Spark / GB10 or Jetson), drivers may report 0 dedicated VRAM even though CUDA works because CPU and GPU share one memory pool. The scheduler treats available system RAM (minus an OS reserve) as schedulable GPU capacity on those hosts so VRAM requirements can still match.
For private runs, confirm your engine is online and workers are configured in Configuration. For cloud runs, queued tasks usually start within seconds unless the platform is under heavy load.
→ Task status and queuing · Tasks API
Engines
An engine manages workers on your hardware.
Install it on your server, and your GPUs become available for tasks.
Route tasks to your hardware with infra: "private" in the API or the Private toggle in the workspace runner. Use infra: "private_first" to prefer your private workers but fall back to cloud when none are available. Agents inherit the same infra setting when they call tools.
| Guide | Topic |
|---|---|
| Why private? | When to use your own workers |
| Installing the engine | Engine setup on Linux and macOS |
| Configuration | Engine and worker settings |
| Managing engines | Drain, update, and restart |
| Using private workers | Workspace and API usage |
Next
Now you know the concepts! Let's build.