Precision (fp16 vs fp32)
The shipped SageMaker image and on-premise packages serve fp16 by default, which is roughly 3× faster than fp32 on the same GPU, for the same workload (benchmarked on an A10G-class GPU); fp32 is an opt-in build. Precision is fixed by the deployed artifact, not a per-request option — see On-premise deployment and AWS SageMaker.Batching (on-premise / self-hosted)
The server auto-batches concurrent requests before running them through the GPU. These env vars control that behavior:
In practice, once the GPU is saturated, tuning these away from their defaults has little effect — they matter mainly if requests are arriving faster than the GPU can drain them. Set them with
-e when starting the container; see Environment variables.

