> ## Documentation Index
> Fetch the complete documentation index at: https://docs.bioptimus.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Performance

> Model-server performance characteristics: what's fixed by the deployment, and what to expect.

For Bioptimus embedding and prediction models, **the GPU forward pass is usually the bottleneck** of a request — a couple of server-level choices dominate throughput. This page covers those. If you're driving inference through the SDK, the client-side knobs (concurrency, tissue masking, cohort workers) are covered in [Performance tuning](/guides/get-started/performance-tuning).

## Precision (fp16 vs fp32)

The single biggest lever is which model variant is deployed: **fp16 is roughly 3× faster than fp32** on the same GPU, for the same workload (benchmarked on an A10G-class GPU). This is a **deployment-time choice**, not a per-request option — it depends on which image/artifact was built and deployed. See [On-premise deployment](/deployment/platforms/on-premise) and [AWS SageMaker](/deployment/platforms/aws-sagemaker) for choosing the fp16 build.

## Batching (on-premise / self-hosted)

The server auto-batches concurrent requests before running them through the GPU. These env vars control that behavior:

| Variable             | Default | Effect                                                               |
| -------------------- | ------- | -------------------------------------------------------------------- |
| `BATCH_MAX_SIZE`     | `48`    | Maximum samples per GPU batch.                                       |
| `BATCH_MAX_WAIT_MS`  | `30.0`  | Maximum time to wait before dispatching a partial batch.             |
| `PREPROCESS_WORKERS` | `8`     | Thread-pool size for request decoding/preprocessing before batching. |

In practice, once the GPU is saturated, tuning these away from their defaults has little effect — they matter mainly if requests are arriving faster than the GPU can drain them. Set them with `-e` when starting the container; see [Environment variables](/deployment/platforms/on-premise#environment-variables).

## GPU-bound behavior

Because the forward pass dominates, throughput scales with how much work is sent to the server — reducing the number of tiles per slide (tissue masking, done client-side) reduces total time proportionally. Once precision and tile count are fixed, the remaining lever is the GPU itself: a bigger/newer GPU raises the ceiling directly, and additional GPUs scale a multi-GPU deployment (1:1 worker-to-GPU mapping).

Seeing lower throughput than this would suggest? See [Troubleshooting](/api-reference/troubleshooting).