Performance tuning

Whole-slide inference is a pipeline: tiles are read and preprocessed on CPU, then run through the model on GPU, then written to disk. For Bioptimus embedding and prediction models, the GPU forward pass is usually the bottleneck — a few knobs matter a lot, and most others matter little. This page covers the client-side (SDK) knobs; for what’s fixed by the deployment (precision, batching env vars), see the Inference Server’s Performance page.

Quick reference

Do this	Expected impact
Use the fp16 checkpoint instead of fp32	Largest single lever — noticeably faster on the same GPU, for the same slide
Tighten tissue masking (`mask_threshold`)	Impact scales directly with tile count — fewer tiles is proportionally less work
Keep concurrency reasonably high (see below)	Too low starves the GPU; past a certain point, more makes no difference
Leave batch size at the default	Has little effect within a normal range — not worth sweeping
Add or upgrade GPUs	The remaining lever once the above are already set

Batch wait time and CPU preprocess worker count rarely matter once the GPU is the bottleneck — leave them at their defaults unless you’ve confirmed with the check in Is the GPU the bottleneck? that the GPU is not saturated.

Backend-specific knobs

Local (in-process)
Remote / AWS SageMaker

from bioptimus.models.backbones import Backbone

model = Backbone(
    "h1",
    backend="local",
    checkpoint=".../h1_optimus_cls_token_fp16.pt2",  # prefer the fp16 checkpoint
    device="cuda:0",
    max_batch_size=128,  # default is a reasonable middle ground; rarely needs changing
)

inferrer.predict(output_path=out, mode="embed", max_concurrency=256)

For a cohort (multiple slides) on a single local GPU, keep Inference(..., workers=...) at 1–2 — all slides share the same GPU, so higher values add threads without adding throughput. To use multiple local GPUs, run one process per GPU (CUDA_VISIBLE_DEVICES=<i>) rather than raising workers. See Cohorts.

max_concurrency is the main lever — size it to your endpoint’s GPU count, not your client’s CPU count:

inferrer.predict(output_path=out, mode="embed", max_concurrency=256)

On a single-GPU endpoint (e.g. ml.g5.xlarge, an A10G-class GPU), lower max_concurrency (e.g. 32) to avoid out-of-memory from too many concurrent requests. On a multi-GPU endpoint, a higher value (e.g. 256) is a reasonable default. See the note in Spatial transcriptomics.

For a cohort, size the client-side workers to a small multiple of the endpoint’s GPU count — too few starves the server’s batching, too many just queues up requests.

Tissue masking (mask_threshold) and tile geometry (size/stride) are backend-agnostic — see Preprocessing tuning knobs. For ready-to-copy starting configurations (single slide vs. cohort), see Recommended starting configurations on the Bioptimus SDK page.

Order-of-magnitude expectations

As a rough reference, on a single mid-range datacenter GPU, embedding a typical whole slide (a few thousand tissue tiles) with the fp16 checkpoint takes on the order of one to a few minutes; fp32 is a few times slower. Tissue coverage and GPU model shift this substantially — measure your own workload (tiles processed ÷ wall-clock time) rather than relying on this as a guarantee.

Slower than expected, or a run that looks stalled? See Common issues.

Overview

Get Started

Preprocessing

Workflows

Troubleshooting

Reference

Quick reference

Backend-specific knobs

​Quick reference

​Backend-specific knobs

Quick reference

Backend-specific knobs