> ## Documentation Index
> Fetch the complete documentation index at: https://docs.bioptimus.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Performance tuning

> What to change to get more embedding/prediction throughput, and what to expect from each change.

Whole-slide inference is a pipeline: tiles are read and preprocessed on CPU, then run through the model on GPU, then written to disk. For Bioptimus embedding and prediction models, **the GPU forward pass is usually the bottleneck** — a few knobs matter a lot, and most others matter little. This page covers the client-side (SDK) knobs; for what's fixed by the deployment (precision, batching env vars), see the Inference Server's [Performance](/api-reference/performance) page.

## Quick reference

| Do this                                      | Expected impact                                                                  |
| -------------------------------------------- | -------------------------------------------------------------------------------- |
| Use the **fp16** checkpoint instead of fp32  | Largest single lever — noticeably faster on the same GPU, for the same slide     |
| Tighten tissue masking (`mask_threshold`)    | Impact scales directly with tile count — fewer tiles is proportionally less work |
| Keep concurrency reasonably high (see below) | Too low starves the GPU; past a certain point, more makes no difference          |
| Leave batch size at the default              | Has little effect within a normal range — not worth sweeping                     |
| Add or upgrade GPUs                          | The remaining lever once the above are already set                               |

Batch wait time and CPU preprocess worker count rarely matter once the GPU is the bottleneck — leave them at their defaults unless you've confirmed with the check in [Is the GPU the bottleneck?](#is-the-gpu-the-bottleneck) that the GPU is *not* saturated.

## Backend-specific knobs

<Tabs>
  <Tab title="Local (in-process)">
    ```python theme={null}
    from bioptimus.models.backbones import Backbone

    model = Backbone(
        "h1",
        backend="local",
        checkpoint=".../h1_optimus_cls_token_fp16.pt2",  # prefer the fp16 checkpoint
        device="cuda:0",
        max_batch_size=128,  # default is a reasonable middle ground; rarely needs changing
    )
    ```

    ```python theme={null}
    inferrer.predict(output_path=out, mode="embed", max_concurrency=256)
    ```

    For a cohort (multiple slides) on a single local GPU, keep `Inference(..., workers=...)` at 1–2 — all slides share the same GPU, so higher values add threads without adding throughput. To use multiple local GPUs, run one process per GPU (`CUDA_VISIBLE_DEVICES=<i>`) rather than raising `workers`. See [Cohorts](/guides/workflows/cohort).
  </Tab>

  <Tab title="Remote / AWS SageMaker">
    `max_concurrency` is the main lever — size it to your endpoint's GPU count, not your client's CPU count:

    ```python theme={null}
    inferrer.predict(output_path=out, mode="embed", max_concurrency=256)
    ```

    <Note>
      On a **single-GPU** endpoint (e.g. `ml.g5.xlarge`, an A10G-class GPU), lower `max_concurrency` (e.g. 32) to avoid out-of-memory from too many concurrent requests. On a **multi-GPU** endpoint, a higher value (e.g. 256) is a reasonable default. See the note in [Spatial transcriptomics](/guides/workflows/spatial-transcriptomics#2-run-inference-image-only).
    </Note>

    For a cohort, size the client-side `workers` to a small multiple of the endpoint's GPU count — too few starves the server's batching, too many just queues up requests.
  </Tab>
</Tabs>

Tissue masking (`mask_threshold`) and tile geometry (`size`/`stride`) are backend-agnostic — see [Preprocessing tuning knobs](/guides/workflows/preprocessing#tuning-knobs).

For ready-to-copy starting configurations (single slide vs. cohort), see [Recommended starting configurations](/guides/get-started/sdk#recommended-starting-configurations) on the Bioptimus SDK page.

<Accordion title="Order-of-magnitude expectations">
  As a rough reference, on a single mid-range datacenter GPU, embedding a typical whole slide (a few thousand tissue tiles) with the fp16 checkpoint takes on the order of one to a few minutes; fp32 is a few times slower. Tissue coverage and GPU model shift this substantially — measure your own workload (tiles processed ÷ wall-clock time) rather than relying on this as a guarantee.
</Accordion>

Slower than expected, or a run that looks stalled? See [Common issues](/guides/get-started/troubleshooting).