Is the GPU actually the bottleneck?
Watch utilization while a run is in progress:- GPU near 100% → you’re GPU-bound. Only precision, tile count, or a bigger/newer GPU will help further — see Performance.
- GPU well below 100% → the GPU is starved. Check the client’s concurrency setting first, then disk/network I/O and CPU load.
INFO level, the server also logs per-batch timing (forward=... cpu_transfer=... total=...). If forward ≈ total and batches fire back-to-back, you’re GPU-bound (expected); a gap between consecutive “batch done” lines means the GPU is waiting on tile production instead.
Failure modes that look like slowness
First request is much slower than the rest
First request is much slower than the rest
Cold start — model loading and first-batch kernel warmup. Don’t benchmark the first request; measure steady-state throughput instead.
Throughput is roughly halved
Throughput is roughly halved
Another process may be sharing the GPU. Check
nvidia-smi for other processes and ensure exclusive access.Raising PREPROCESS_WORKERS makes things worse
Raising PREPROCESS_WORKERS makes things worse
Oversubscribing CPU cores causes thread thrashing. Don’t set it above the host’s core count; it rarely helps once the GPU is the bottleneck anyway.
Model fails to load with a zip/extraction error
Model fails to load with a zip/extraction error
.pt2 artifacts extract to a temp directory (TMPDIR, default /tmp) on load; a failed load can leave stale extraction directories behind and fill the disk. Check df -h /tmp, clear stale extraction directories, or point TMPDIR at a larger volume.Errors
503 and out-of-memory responses usually mean too many requests are arriving concurrently for the deployed batch size — see Errors for the full status code reference.
Calling the server through the SDK? For connection issues (e.g.
/ping not responding), see the SDK’s Common issues guide.