# Best Practices

> Performance optimization and coding patterns for inference shell apps.

**URL:** https://inference.sh/docs/extend/best-practices
**Last updated:** 2026-10-12

---

Performance optimization and coding patterns for inference shell apps.

---

## Device Management

**Never hardcode "cuda"** - use accelerate for automatic device detection. `accelerate` is not preinstalled; add it to `requirements.txt`:

```python
from accelerate import Accelerator

class App(BaseApp):
    async def setup(self, config):
        self.accelerator = Accelerator()
        self.device = self.accelerator.device
```

---

## Model Loading

Use HuggingFace hub for downloads. Add `huggingface_hub` to `requirements.txt` (and `hf_transfer` if you set `HF_HUB_ENABLE_HF_TRANSFER`):

```python
import os
from huggingface_hub import snapshot_download

os.environ["HF_HUB_ENABLE_HF_TRANSFER"] = "1"

class App(BaseApp):
    async def setup(self, config):
        self.model_path = snapshot_download(
            repo_id="org/model-name",
            resume_download=True,
        )
```

**Avoid**:
- Hardcoded local directories (`local_dir="./models"`)
- Subprocess calls to `huggingface-cli`
- Assuming specific file structures

---

## Memory Management

### Clean Up GPU Memory

```python
import torch
import gc

def cleanup_memory():
    if torch.cuda.is_available():
        torch.cuda.empty_cache()
    gc.collect()

async def run(self, input_data):
    result = self.model(input_data)
    cleanup_memory()
    return result
```

### Use Mixed Precision

```python
model = model.to(dtype=torch.bfloat16)

# Or with autocast
from torch.amp import autocast
with autocast('cuda'):
    output = model(input)
```

---

## Error Handling

<div data-type="codegroup">

```python Python
import logging

async def run(self, input_data):
    try:
        result = self.process(input_data)
        return AppOutput(result=result)
    except Exception as e:
        logging.error(f"Processing failed: {e}")
        raise ValueError(f"Failed to process: {str(e)}")
```

```javascript Node.js
async run(inputData) {
  try {
    const result = await this.process(inputData);
    return { result };
  } catch (e) {
    console.error(`Processing failed: ${e.message}`);
    throw new Error(`Failed to process: ${e.message}`);
  }
}
```

</div>

---

## File Handling

<div data-type="codegroup">

```python Python
import tempfile

async def run(self, input_data):
    with tempfile.NamedTemporaryFile(suffix='.jpg', delete=False) as tmp:
        output_path = tmp.name
    save_image(result, output_path)
    return AppOutput(image=File(path=output_path))
```

```javascript Node.js
import { tmpdir } from "node:os";
import { join } from "node:path";

async run(inputData) {
  const outputPath = join(tmpdir(), "output.jpg");
  await saveImage(result, outputPath);
  return { image: { path: outputPath } };
}
```

</div>

---

## Flash Attention

```python
model = AutoModel.from_pretrained(
    "model-name",
    attn_implementation="flash_attention_2",
    torch_dtype=torch.bfloat16
)
```

---

## Pre-deployment Checklist

- [ ] All imports work correctly
- [ ] `setup()` loads models successfully
- [ ] `run()` processes test input
- [ ] No hardcoded paths or devices
- [ ] Proper error handling
- [ ] Memory cleaned up after requests

---

## Next

→ [Output metadata](/docs/extend/output-meta) — usage fields for billing  
→ [Usage-based pricing](/docs/extend/pricing) — CEL formulas