Common issues

When a run fails or behaves unexpectedly, two tools narrow it down before you read a traceback:

Gate a deployment before the expensive run with bioptimus verify — one synthetic tile, checked against the model contract.
Self-diagnose the environment with snapshot(), or run a single probe_* for a fast, targeted check. Most issues below map to one probe — the same evidence a support bundle carries.

Every SDK error also carries a code and a one-line fix; the full list is in Error codes.

Server not responding? Check /ping first

Before constructing a Backbone (or an Inference pipeline), confirm the on-premise server is reachable:

import requests

requests.get("http://localhost:8080/ping", timeout=5).json()
# {"status": "ok", "models": ["h1", "tissue-seg"]}

If this raises ConnectionError / connection refused or times out, nothing is serving at that URL:

Not started — launch the container, mapping port 8080, then wait for models to load. See On-premise deployment:
docker run -d --name bioptimus-server --gpus all -p 8080:8080 <image> serve
503 {"status": "loading"} — models are still initialising; wait and retry.
Wrong URL/port — base_url (and the pipeline’s api_url) must be the server’s host and port, with no trailing /ping.

For SageMaker there is no /ping: confirm the endpoint is InService and that endpoint_name / region_name are correct.

CUDA not available, or inference falls back to CPU

The SDK loaded but the GPU is not being used. Check what the runtime actually sees:

from bioptimus.observability.probes import probe_gpu

gpu = probe_gpu()
gpu["cuda_available"]   # False -> CUDA is not usable here
gpu.get("reason")       # why: PyTorch missing, or CUDA cannot be queried

cuda_available is False on a GPU host — the NVIDIA driver is usually too old for the CUDA build PyTorch ships. probe_gpu reports the driver and CUDA build versions side by side even when CUDA is down.
A device loads but kernels fail at launch — its compute capability is missing from PyTorch’s compiled build (common after a wheel reinstall); probe_gpu flags that device.
Running on a CPU-only box — move to a GPU instance. This surfaces as GPU_UNAVAILABLE.

Model loads but fails on the first tile

A model that loads and then fails at inference (MODEL_FORWARD_FAILED) is almost always an architecture or dependency mismatch, not a problem with the slide. Two probes pinpoint it:

from bioptimus.observability.probes import probe_pt2_compat, probe_dependencies

probe_pt2_compat({"h1": "h1.pt2"})[0]["compatible"]   # False -> built for another GPU arch
probe_dependencies()["conflicts"]                       # non-empty -> a dependency is out of range

.pt2 architecture mismatch — the AOTInductor artifact is compiled for a specific GPU/CPU architecture. probe_pt2_compat reports compiled_for versus the host and the reason for any mismatch.
Dependency drift — upgrading a pinned dependency past its supported ceiling (for example zarr>=3 or a too-new torch) breaks the compiled artifacts silently. probe_dependencies reports each conflict as {name, installed, required}.

ImportError: a backend needs an optional extra

Some backends ship as optional extras to keep the base install light. Using one without its extra raises BioptimusImportError (a real ImportError subclass) tagged DEPENDENCY_MISSING, with the exact command in its remediation:

Feature	Install
`local` backend / PyTorch inference	`pip install bioptimus-sdk[torch]`
`huggingface` backend	`pip install bioptimus-sdk[huggingface]`
GPU-accelerated WSI decoding (cuCIM)	`pip install bioptimus-sdk[cucim]`

OpenSlide and tiffslide readers are bundled by default. If an import still fails after installing, probe_host() catches the “installed it but it’s not found” case — check executable and in_venv to confirm you are running the interpreter you installed into.

A slide won't open

When a WSI fails to open, first check which readers actually imported on this host:

from bioptimus.observability.probes import probe_wsi_backends

probe_wsi_backends()
# {"openslide": {"available": True, "version": "..."},
#  "cucim": {"available": False, "reason": "..."}, ...}

A backend reporting available: False carries the import reason — usually a missing native library. Install the matching extra (above) or convert the slide to a format one of the available backends reads.

.pt2 fails to load, or the temp directory fills up

Loading a .pt2 extracts it into the system temp directory; if that directory is full or read-only, extraction fails (DISK_FULL). probe_disk reports the headroom and flags both failure modes:

from bioptimus.observability.probes import probe_disk

d = probe_disk()
d["warning"]                 # "TMP_NOT_WRITABLE", "TMP_NEARLY_FULL", or None
d["aotinductor_temp_dirs"]   # leftover extractions pile up when packages aren't cleaned

Point TMPDIR at a writable volume with room, and clear leaked aotinductor extraction directories if the count keeps climbing.

A run looks stuck near the end

The writer is finalizing outputs (thumbnail, tissue mask, metadata) — this is expected and brief. A run that stays stuck with an idle GPU points to I/O, not the model; see the Inference Server’s Troubleshooting guide.

Server responding fine but throughput lower than expected, or a run that looks stalled? That’s covered in the Inference Server’s Troubleshooting guide. For what to change to get more throughput, see Performance tuning.

Need to share details with support? See Logging and Support bundles.

Overview

Get Started

Preprocessing

Workflows

Troubleshooting

Reference