> ## Documentation Index
> Fetch the complete documentation index at: https://docs.bioptimus.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Performance tuning

> What to change to get more embedding/prediction throughput, and what to expect from each change.

Whole-slide inference is a pipeline: tiles are read and preprocessed on CPU, then run through the model on GPU, then written to disk. For Bioptimus embedding and prediction models, **the GPU forward pass is usually the bottleneck** — a few knobs matter a lot, and most others matter little. This page covers the client-side (SDK) knobs; for what's fixed by the deployment (precision, batching env vars), see the Inference Server's [Performance](/api-reference/performance) page.

## Quick reference

| Do this | Expected impact |
| - | - |
| Use the **fp16** checkpoint instead of fp32 | Largest single lever — noticeably faster on the same GPU, for the same slide |
| Tighten tissue masking (`mask_threshold`) | Impact scales directly with tile count — fewer tiles is proportionally less work |
| Keep concurrency reasonably high (see below) | Too low starves the GPU; past a certain point, more makes no difference |
| Leave batch size at the default | Has little effect within a normal range — not worth sweeping |
| Need both outputs? Use `predict_with_embedding` instead of separate `predict()`/`embed()` calls | M-Optimus only — halves GPU forward passes for the same tiles (one backbone run instead of two) |
| Add or upgrade GPUs | The remaining lever once the above are already set |

Batch wait time and CPU preprocess worker count rarely matter once the GPU is the bottleneck — leave them at their defaults unless you've confirmed with the check in [Is the GPU the bottleneck?](/api-reference/troubleshooting#is-the-gpu-actually-the-bottleneck) that the GPU is *not* saturated.

For a worked example of the combined mode, see [Get embedding + prediction together](/guides/workflows/spatial-transcriptomics#6-get-embedding-prediction-together-optional) in Spatial transcriptomics.

## Backend-specific knobs

<Tabs>
  <Tab title="Local (in-process)">
    ```python theme={null}
    from bioptimus.models.backbones import Backbone

    model = Backbone(
        "h1",
        backend="local",
        checkpoint=".../h1_optimus_cls_token_fp16.pt2",  # prefer the fp16 checkpoint
        device="cuda:0",
        max_batch_size=128,  # default is a reasonable middle ground; rarely needs changing
    )
    ```

    ```python theme={null}
    inferrer.predict(output_path=out, mode="embed", max_concurrency=256)
    ```

    For a cohort (multiple slides) on a single local GPU, keep `Inference(..., workers=...)` at 1–2 — all slides share the same GPU, so higher values add threads without adding throughput. To use multiple local GPUs, run one process per GPU (`CUDA_VISIBLE_DEVICES=<i>`) rather than raising `workers`. See [Cohorts](/guides/workflows/cohort).
  </Tab>

  <Tab title="Remote / AWS SageMaker">
    `max_concurrency` is the main lever — size it to your endpoint's GPU count, not your client's CPU count:

    ```python theme={null}
    inferrer.predict(output_path=out, mode="embed", max_concurrency=256)
    ```

    <Note>
      On a **single-GPU** endpoint (e.g. `ml.g5.xlarge`, an A10G-class GPU), lower `max_concurrency` (e.g. 32) to avoid out-of-memory from too many concurrent requests. On a **multi-GPU** endpoint, a higher value (e.g. 256) is a reasonable default. See the note in [Spatial transcriptomics](/guides/workflows/spatial-transcriptomics#2-run-inference-image-only).
    </Note>

    For a cohort, size the client-side `workers` to a small multiple of the endpoint's GPU count — too few starves the server's batching, too many just queues up requests.
  </Tab>
</Tabs>

Tissue masking (`mask_threshold`) and tile geometry (`size`/`stride`) are backend-agnostic — see [Preprocessing tuning knobs](/guides/workflows/preprocessing#tuning-knobs).

For ready-to-copy starting configurations (single slide vs. cohort), see [Recommended starting configurations](/guides/get-started/sdk#recommended-starting-configurations) on the Bioptimus SDK page.

<Accordion title="Order-of-magnitude expectations">
  As a rough reference, on a single mid-range datacenter GPU, embedding a typical whole slide (a few thousand tissue tiles) with the fp16 checkpoint takes on the order of one to a few minutes; fp32 is a few times slower. Tissue coverage and GPU model shift this substantially — measure your own workload (tiles processed ÷ wall-clock time) rather than relying on this as a guarantee.
</Accordion>

Slower than expected, or a run that looks stalled? See [Common issues](/guides/get-started/troubleshooting).


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.