Skip to main content
For Bioptimus embedding and prediction models, the GPU forward pass is usually the bottleneck of a request — a couple of server-level choices dominate throughput. This page covers those. If you’re driving inference through the SDK, the client-side knobs (concurrency, tissue masking, cohort workers) are covered in Performance tuning.

​
Precision (fp16 vs fp32)

The single biggest lever is which model variant is deployed: fp16 is roughly 3× faster than fp32 on the same GPU, for the same workload (benchmarked on an A10G-class GPU). This is a deployment-time choice, not a per-request option — it depends on which image/artifact was built and deployed. See On-premise deployment and AWS SageMaker for choosing the fp16 build.

​
Batching (on-premise / self-hosted)

The server auto-batches concurrent requests before running them through the GPU. These env vars control that behavior: In practice, once the GPU is saturated, tuning these away from their defaults has little effect — they matter mainly if requests are arriving faster than the GPU can drain them. Set them with -e when starting the container; see Environment variables.

​
GPU-bound behavior

Because the forward pass dominates, throughput scales with how much work is sent to the server — reducing the number of tiles per slide (tissue masking, done client-side) reduces total time proportionally. Once precision and tile count are fixed, the remaining lever is the GPU itself: a bigger/newer GPU raises the ceiling directly, and additional GPUs scale a multi-GPU deployment (1:1 worker-to-GPU mapping). Seeing lower throughput than this would suggest? See Troubleshooting.