# Troubleshooting

> Common issues and solutions for inference shell apps.

**URL:** https://inference.sh/docs/extend/troubleshooting
**Last updated:** 2026-10-12

---

Common issues and solutions for inference shell apps.

---

## Task stuck in Queued

**Symptom:** A run stays in **Queued** and never moves to **Running**.

**Check:**

1. **Cloud run** — Usually resolves when a cloud worker is free. Heavy load or maintenance can add a short wait.
2. **Private run** — Open [Engines](https://app.inference.sh/engines). The engine must be online and workers must match the app's GPU/CPU requirements. See [Using private workers](/docs/private/using) and [Configuration](/docs/private/config).
3. **Resource mismatch** — The app may require more VRAM or a specific GPU type than any connected worker provides.
4. **Infra mismatch** — Some apps only allow cloud or only private; wrong `infra` fails immediately instead of queuing.

Tasks wait in the queue for up to roughly **24 hours** before failing with `task expired while waiting for available capacity, please retry`. If no worker matches the app's resource requirements at all, failure happens sooner (~10 minutes) with `no active workers with matching resources`. See [Task status](/docs/concepts/tasks#queued-longer-than-expected) and [Workers](/docs/concepts/workers#when-no-worker-is-available-yet).

**CLI:** Use `belt task get <id> -f` to watch status until the task starts or fails. See [CLI setup](/docs/extend/cli-setup#tasks-cli).

---

## Platform task failures

**Symptom:** Task status is **Failed** with a short, generic error like `task interrupted, please retry` or `task could not be dispatched, please retry`.

**Cause:** The platform hit a transient infrastructure issue — worker disconnect, engine reconnect, dispatch retry exhaustion, or similar. These messages are intentionally user-friendly; internal diagnostics are not exposed in the `error` field.

**What to do:**

1. **Retry the run**: most platform failures are transient.
2. **Check task logs**: app-level errors (validation, OOM, model failures) appear with your app's own message and are not sanitized.
3. **Session runs**: if you see `session worker busy too long`, wait for the prior call on that session to finish or serialize session calls. See [Sessions: task failures](/docs/extend/sessions#task-failures-session-scheduling).
4. **Secrets**: `failed to prepare secrets for this app, please check your secret configuration` means a required workspace secret is missing or invalid. Add or fix secrets in [vaults → secrets](https://app.inference.sh/vaults?tab=secrets).

For the full list of platform error strings, see [Task failure messages](/docs/concepts/tasks#failed-tasks-and-error-messages).

---

## Access denied (IP blocked)

**Symptom:** Every API request returns HTTP **403** with:

```json
{
  "message": "Access denied. This attempt has been logged. If you believe this is a mistake, please disconnect any VPN or proxy and try again, or contact security@inference.sh."
}
```

**Cause:** Your IP address was blocked for abusive or malicious activity. The check runs before authentication on almost every route. Some health and status endpoints stay reachable.

**What to do:**

1. Disconnect VPN or proxy services and retry from a direct connection.
2. If the block persists, contact [security@inference.sh](mailto:security@inference.sh) with your public IP address and what you were trying to do.

This is separate from permission **403** errors (`forbidden`, `feature_not_available`) and from magic-link rate limits (**429**). See [REST overview: Pre-authentication access controls](/docs/api/rest/overview#pre-authentication-access-controls).

---

## Service not available in your region

**Symptom:** Every API request returns HTTP **451** with plain text:

```
service not available in your region
```

**Cause:** inference shell blocks API access from countries and regions under OFAC sanctions. The check runs before authentication on almost every route. Blocked countries are Cuba (CU), Iran (IR), North Korea (KP), Russia (RU), and Syria (SY). Within Ukraine, Crimea, Sevastopol, Donetsk, and Luhansk are also blocked.

**What to do:**

1. Disconnect VPN or proxy services that exit in a blocked country or region and retry from a permitted location.
2. If you believe the geo lookup is wrong, contact [security@inference.sh](mailto:security@inference.sh) with your public IP address and location.

This is separate from IP abuse blocks (**403**) and from plan-limit errors. See [REST overview: Pre-authentication access controls](/docs/api/rest/overview#pre-authentication-access-controls).

---

## Signup blocked (disposable email)

**Symptom:** Sign-up fails with **Temporary email addresses are not allowed. Please use a permanent email address.** API responses use code `disposable_email` (HTTP **400**) on magic link send and OAuth callbacks.

**Cause:** inference shell blocks new accounts on known disposable or temporary email domains. The check runs when a new user record would be created, not when an existing account signs in.

**What to do:**

1. Sign up with a permanent email address.
2. If your account was registered on such a domain before the restriction, sign in with your original method. Existing accounts are not blocked.
3. On a first SAML login, the same rejection surfaces as a generic `user_creation_failed` error instead of `disposable_email`. Retry with a permanent address.

**Plus (`+`) aliases:** Magic link sign-up currently allows `+` tags (for example `you+tag@example.com`). If you later see `invalid_email` on magic link send, the platform may have turned on plus-alias blocking for new accounts on your email provider—use your base address, or sign in with Gmail/Googlemail where `+` tags fold into one mailbox. Existing accounts are not affected.

→ [REST overview: Signup email restrictions](/docs/api/rest/overview#signup-email-restrictions) · [Magic link errors](/docs/api/rest/magic-link#errors)

---

## CLI billing or plan limit errors

**Symptom:** `belt` or `infsh` prints a message about insufficient balance, upgrading your plan, or a link to billing settings.

**Cause:** The API returned **402 Payment Required** (credits) or **403 Forbidden** with an entitlement error (`upgrade_available` in the error metadata).

**What to do:**

1. **Insufficient balance**: add credits at [settings → workspace → billing](https://app.inference.sh/settings/team/billing). The CLI includes the billing URL in the error message.
2. **Plan limit**: upgrade at [settings → workspace → subscription](https://app.inference.sh/settings/team/billing/subscription) when the error links to the subscription page.
3. **Addon entitlement**: subscribe at [settings → workspace → add-ons](https://app.inference.sh/settings/team/billing/addons) when the error links to the add-ons page (for example a blocked `feature:seedance` store app).
4. **Not logged in**: run `belt login` (or `infsh login`) if you see "not logged in or session expired".

These messages come from the API error body; other failures use [RFC 9457 problem+json](/docs/api/rest/overview#error-responses).

---

## Deploy: `integrations:` in `inf.yml`

**Symptom:** Loading or deploying fails with `inf.yml: the integrations: key was renamed credentials: — rename it and deploy again`.

**Cause:** CLI v1.18.59 and later no longer read the old `integrations:` block. Requirements must live under `credentials:`.

**What to do:**

1. Rename the top-level key to `credentials:` (same list shape).
2. Run `belt app credentials list` to confirm capability `key` values.
3. Redeploy.

See [Credentials — Declaring credentials](/docs/extend/credentials#declaring-credentials).

---

## Deploy: `admin_session_required` or another namespace

**Symptom:** `belt app deploy` fails with HTTP 403 and a message that starts `Deploying into namespace "…" needs admin access.`

**Cause:** `inf.yml` names a namespace other than your active workspace, and your account has the platform staff role. Deploying into another namespace needs admin access; the deploy is refused rather than moved into your workspace. The error code says what is missing:

| Code | What to do |
|---|---|
| `admin_session_required` | Run `belt admin elevate` and approve it in the browser, then deploy again |
| `admin_scopes_not_approved` | This login was not approved with admin scopes. Run `belt login` again and approve them, then `belt admin elevate` |
| `admin_authenticator_required` / `otp_required` | Add an authenticator app under Settings > Security, or confirm with it, then retry |
| `forbidden` | Your account has the staff role but not admin access |

To deploy into your own workspace, set `namespace` to your workspace or remove it. A member without the staff role who names another namespace deploys into their own workspace.

→ [Deploying: Namespace](/docs/extend/deploying#namespace)

---

## Deploy: unknown credential keys

**Symptom:** `belt app deploy` fails during credential validation with `unknown credential keys: [...]` and suggests `belt app credentials list`.

**Cause:** A `credentials:` entry includes a `key` that is not a registered platform capability. Capability keys must match the list from `belt app credentials list` (or `GET /integrations/capabilities`).

**What to do:**

1. Fix typos in `key` (for example `google.sheet` vs `google.sheets`).
2. For BYOK API keys tied to a provider credential, use a **provider-only** entry instead of inventing a capability name:

   ```yaml
   credentials:
     - provider: acme
       secrets: [ACME_API_KEY]
   ```

   Deploy does not treat provider-only rows as capabilities; the API checks provider requirements when the version is created. See [Credentials — Provider-only entries](/docs/extend/credentials#api-keys-from-a-provider).
3. If you set both `provider` and `key`, only `key` is validated as a capability—the value must still appear in `belt app credentials list`.

---

## Tasks rejected during engine maintenance

**Symptom:** New private tasks fail immediately or stay queued while you update or drain an engine.

**Cause:** While an engine is **draining** (or receiving an **update**), it rejects new work with reason `engine_draining` until in-flight tasks finish and the engine restarts.

**What to do:**

1. Wait for running tasks on that engine to complete, or cancel them if you need a faster turnaround.
2. Use **Drain** before maintenance and **Update** when you want a safe binary upgrade — see [Engines API](/docs/api/rest/engines).
3. Route urgent work to cloud infra or another online engine with `infra: "cloud"` or a different private engine.

---

## Plan limits and billing errors

**Symptom:** API or CLI calls fail with HTTP **402** or **403**, or messages about limits, balance, or upgrading your plan.

### Plan usage limit (`limit_exceeded`, HTTP 402)

You hit a **subscription limit** — for example too many API keys, connectors, knowledge bases, private apps, or storage.

**What to do:**

1. The web app opens an **upgrade modal** with a **recommended** plan, the cheapest tier that lifts the blocked limit. It shows current usage vs your cap when the cap is greater than zero.
2. Open [settings → workspace → subscription](https://app.inference.sh/settings/team/billing/subscription) to compare all plans and change your subscription.
3. Delete or reduce usage (unused API keys, old knowledge entries, large files).
4. Inspect limits programmatically: `GET /entitlements` for limits and `GET /entitlements/usage` for current counts — see [Entitlements API](/docs/api/rest/entitlements).

### Feature not on your tier (`feature_not_available`, HTTP 403)

The operation requires a capability your plan does not include (for example `feature:publish_apps` or `feature:webhooks`).

The web app's upgrade modal lists higher tiers and recommends the cheapest plan that enables the feature. Upgrade at [settings → workspace → subscription](https://app.inference.sh/settings/team/billing/subscription) or contact support if you believe the feature should already be enabled.

### Insufficient balance (`payment_required`, HTTP 402)

Task and agent runs charge **prepaid credits**. When balance is too low, `POST /run` and agent message endpoints return `payment_required`. They also return it when the payer's billing `status` (`GET /billing`) is not `active`: `spend_locked`, `manual_hold` or `suspended`. Those statuses refuse free runs too.

Add credits at [settings → workspace → billing](https://app.inference.sh/settings/team/billing). A credit that takes the balance above the spend floor clears `spend_locked`; `manual_hold` and `suspended` need support. See [Billing: Get billing account](/docs/api/rest/billing#get-billing-account).

### Payment method required (`payment_method_required`, HTTP 402)

**Symptom:** A run of an app that costs nothing for your input returns `payment_method_required` while the balance is too low for a paid run.

**Cause:** A free run needs no credits, but the workspace that pays must have a saved payment method and billing `status` `active`. The response's `meta.billing_page` is `/settings/team/billing/payments`.

**What to do:** Save a card under [settings → workspace → billing → payments](https://app.inference.sh/settings/team/billing/payments) and retry. Nothing is charged for a free run. See [Billing: free runs](/docs/api/rest/billing#free-runs).

### Entitlement lookup failed (`entitlement_unavailable`, HTTP 500)

**Symptom:** A request that checks plan limits (for example `POST /files`, `GET /entitlements`, or `GET /entitlements/usage`) fails with `could not verify plan limits, try again` and code `entitlement_unavailable`.

**Cause:** The API could not read your workspace's entitlement rows. This is a transient server error, not a plan cap or a disabled feature. The API fails closed: it denies the operation until entitlements can be verified again.

**What to do:**

1. Retry the request after a short delay.
2. If it persists, contact support. Upgrading your plan or adding credits does not fix it.

### CLI messages

The **belt** / **infsh** CLI prints the API `detail` message for most errors. When the response includes `upgrade_available: true` in the entitlement metadata under top-level `meta`, the CLI appends an upgrade URL. When `addon_plan_id` is also present, the URL points to the add-ons page; otherwise it points to the subscription page. For HTTP **402** without that metadata, the CLI suggests the billing URL for credits instead. Check whether the failure is a **plan limit** (entitlements), **addon** (feature gate), or **balance** (credits).

→ [REST overview — Billing and plan limits](/docs/api/rest/overview#billing-and-plan-limits) · [Entitlements API](/docs/api/rest/entitlements)

---

## Import Errors

### "ModuleNotFoundError" in Production

**Common cause**: a package your code imports is not in `requirements.txt`. The Python base environment holds only build tools, `pydantic` and `inferencesh`; `torch`, `accelerate`, `huggingface_hub`, `transformers` and `diffusers` are not preinstalled. A version that ran before may fail once it is set up again on a new worker, or when you deploy a new version. See [Python environment](/docs/extend/configuration#python-environment).

**Solutions**:

1. Add the missing package to `requirements.txt` and deploy again. `transformers` and `diffusers` need `accelerate` for `device_map`, `low_cpu_mem_usage` and `enable_model_cpu_offload()`.

2. Add `__init__.py` files to all packages

3. Add current directory to Python path:
```python
import sys, os
sys.path.append(os.path.dirname(os.path.abspath(__file__)))
```

4. For local packages, use editable installs:
```txt
-e ./local_package_directory
```

---

## Memory Issues

### "CUDA out of memory"

**Solutions**:

1. Reduce batch size
2. Use mixed precision: `model.to(dtype=torch.float16)`
3. Enable gradient checkpointing: `model.gradient_checkpointing_enable()`
4. Clear cache: `torch.cuda.empty_cache()`
5. Increase VRAM in `inf.yml`

### Memory Leaks

Clean up after each request:
```python
import gc, torch

async def run(self, input_data):
    result = self.process(input_data)
    if torch.cuda.is_available():
        torch.cuda.empty_cache()
    gc.collect()
    return result
```

---

## Device Errors

### "Expected all tensors to be on the same device"

Ensure all tensors are on the same device:
```python
input_tensor = input_tensor.to(self.device)
```

### "CUDA not available"

1. Check `inf.yml` GPU requirements:
```yaml
resources:
  gpu:
    count: 1
    vram: 24  # 24GB
```

2. Use device detection (add `accelerate` to `requirements.txt`):
```python
from accelerate import Accelerator
device = Accelerator().device
```

---

## Model Loading Errors

### "File not found" After Download

Don't assume file paths:
```python
model_path = snapshot_download(repo_id="org/model")
config_path = os.path.join(model_path, "config.yaml")
if os.path.exists(config_path):
    # Load config
```

### "Token required for gated model"

Add HF_TOKEN to secrets:
```yaml
secrets:
  - key: HF_TOKEN
    description: HuggingFace token for gated models
```

---

## File Path Issues

### Temporary Files Deleted Too Early

Use `delete=False`:
```python
with tempfile.NamedTemporaryFile(suffix='.jpg', delete=False) as tmp:
    output_path = tmp.name
```

### Path Separators

Use `os.path.join`:
```python
# ✅ Good
path = os.path.join("models", "config", "settings.json")
```

---

## Dependency Issues

### Version Conflicts

Pin compatible versions:
```txt
torch==2.6.0
numpy>=1.23.5,<2
```

### Flash Attention Build Errors

Use pre-built wheels in `requirements2.txt`

---

## Debug Mode

Add logging:
```python
import logging
logging.basicConfig(level=logging.DEBUG)

async def setup(self, config):
    logging.debug(f"Config: {config}")
    logging.info("Starting model load...")
```

---

## Node.js-Specific Issues

### "ERR_MODULE_NOT_FOUND"

**Importing `@inferencesh/sdk` in plain Node:** up to **v0.22.1** the package's `dist/` used extensionless relative imports, so `import '@inferencesh/sdk'` and most subpaths failed with `ERR_MODULE_NOT_FOUND` without a bundler. Upgrade to **@inferencesh/sdk ≥ v0.23.0**.

For your own ESM project files, ensure `"type": "module"` is in your `package.json` and use file extensions in relative imports:
```javascript
import { helper } from "./helper.js"; // .js required for ESM
```

### "Cannot use import statement outside a module"

Your `package.json` must have:
```json
{
  "type": "module"
}
```

### Native Module Build Errors

Some packages (e.g., `sharp`, `canvas`) need system libraries. Add them to `packages.txt`:
```
libvips-dev
```

---

## Claude Code belt plugin issues

**Symptom:** Hooks stop running, `belt suggest` never appears in context, or Claude Code reports that the plugin directory does not exist.

**Check:**

```bash
belt plugin doctor
```

The doctor verifies belt CLI availability, authentication (not guest), the belt entry in Claude's `installed_plugins.json`, that the install path exists, that `hooks/hooks.json` is present, that hook scripts in `bin/` use the thin shim pattern (`exec belt plugin hook <event>`) across Claude and Codex plugin caches, and recent hook activity in `~/.belt/hooks.log`.

**Fix:**

```bash
belt plugin doctor --fix
```

`--fix` reinstalls the belt plugin when the cache path is stale or missing (`claude plugin uninstall belt@belt-sh-skills`, then `claude plugin install belt`, with marketplace add as fallback), removes other stale cache directories under the plugin install parent, and patches hook scripts that still contain embedded logic instead of the thin shim format (`exec belt plugin hook <event>`). The same hook patching runs automatically after `belt plugin install claude` because marketplace caching can serve stale hook scripts. For auth issues it prints `belt login` guidance (guest sessions disable knowledge persistence).

When all checks pass, doctor prints **All checks passed**. When `--fix` applies repairs, it lists each change (for example `fixed: removed stale cache dir abc123`, `fixed: reinstalled plugin (cache path was stale)`, or `fixed: patched hook-stop.sh in <hash>`), then prompts you to restart. Inside a Claude Code session, doctor suggests **Run /reload-plugins or restart this session to apply changes**; from a regular terminal, it suggests **Restart your agent session to apply changes**.

→ [Claude Code plugin](/docs/extend/coding-agents#claude-code-plugin) · [CLI setup — Agent integration](/docs/extend/cli-setup#agent-integration-cli)

---

## Kiro belt hooks not running

**Symptom:** `belt suggest` never appears in Kiro sessions after `belt plugin install kiro`.

**Check:**

```bash
belt plugin doctor
```

Look for the `kiro-agent` check. Belt's hooks live in `~/.kiro/agents/belt.json` and run only while **belt** is Kiro's default agent (`chat.defaultAgent` in `.kiro/settings/cli.json`; the workspace setting wins over `~/.kiro/settings/cli.json`).

**Fix:**

```bash
belt plugin doctor --fix           # sets belt as default when no other agent is selected
kiro-cli agent set-default belt    # or set it yourself
```

If another Kiro agent is your default, copy the `hooks` block from `~/.kiro/agents/belt.json` into that agent's config instead.

→ [Coding agents: Other agent runtimes](/docs/extend/coding-agents#other-agent-runtimes-belt-plugin-install)

---

## OpenCode or Kilo: no belt suggestions on prompt

**Symptom:** `belt suggest` never appears in OpenCode or Kilo sessions after `belt plugin install`, or the same suggestions repeat several times per message.

**Cause:** OpenCode and Kilo inject belt output through a system-prompt transform rather than a direct prompt hook. Belt v1.18.43 and later pass the user prompt into the hook correctly; older plugin installs ran the hook without stdin, so belt received no prompt and printed nothing.

**Fix:**

```bash
belt update                        # belt v1.18.43+ required
belt plugin install opencode       # or: belt plugin install kilo
```

Restart OpenCode or Kilo after reinstalling. Verify with `belt plugin status` — the agent should show belt hooks at user or project scope.

→ [Coding agents: Other agent runtimes](/docs/extend/coding-agents#other-agent-runtimes-belt-plugin-install) · [CLI setup — Agent integration](/docs/extend/cli-setup#agent-integration-cli)

---

## Goose: belt skills missing from `goose skills list`

**Symptom:** `goose skills list` omits belt skills, but belt hooks still run.

**Cause:** A `plugin.json` at `~/.agents/plugins/belt/plugin.json` (hand-written or from an older workaround). Goose hides plugin components when manifest paths are wrong — no error, zero exit.

**Fix:**

```bash
rm ~/.agents/plugins/belt/plugin.json   # only if you added one manually
belt plugin install goose               # reinstall skills and hooks
goose skills list
```

If you installed before September 2026, hooks may still be under `~/.agents/plugins/belt-test/` — `belt plugin install goose` moves them to `~/.agents/plugins/belt/`.

→ [Coding agents: Goose](/docs/extend/coding-agents#other-agent-runtimes-belt-plugin-install)

---

## Stale CLI on PATH

**Symptom:** `belt` or `infsh` fails with a Python import error, runs an old version, or behaves differently depending on which command name you use.

**Cause:** An older install shadows the current binary. The usual culprit is the legacy PyPI `inferencesh` package, which shipped an `infsh` launcher.

**Fix:**

```bash
belt auth doctor            # look for a warning on the path check
pip uninstall inferencesh   # if you installed the old PyPI package
which belt infsh inferencesh
```

Remove the stale binary, or put the directory named in the `path` check's hint earlier in `PATH`. See [CLI setup: Authentication (`belt auth doctor`)](/docs/extend/cli-setup#authentication-belt-auth-doctor).

---

## CLI billing and authentication errors

**Symptom:** `belt` or `infsh` fails with a message about balance, plan limits, or login.

| Message | What to do |
|---------|------------|
| `insufficient balance` + billing URL | Add credits at [settings → workspace → billing](https://app.inference.sh/settings/team/billing) |
| `upgrade at:` + subscription URL | Raise plan limits at [settings → workspace → subscription](https://app.inference.sh/settings/team/billing/subscription) |
| `upgrade at:` + add-ons URL | Subscribe at [settings → workspace → add-ons](https://app.inference.sh/settings/team/billing/addons) |
| `not logged in or session expired` | Run `belt login` (or `infsh login`) |

See [CLI setup — Billing and plan limits](/docs/extend/cli-setup#billing-and-plan-limits) for details.

---

## Next

→ [Best Practices](/docs/extend/best-practices) - Optimization patterns