> ## Documentation Index
> Fetch the complete documentation index at: https://docs.bioptimus.com/llms.txt
> Use this file to discover all available pages before exploring further.

# bioptimus.runtime.batch_scheduler

Dynamic request batching for GPU inference.

Implements a two-stage pipeline:

1. **Deserialization** (parallel, per-request): Each incoming request is decoded, transformed, and validated in a thread pool.  Failures are returned immediately as per-request errors (HTTP 400) without entering the batch queue.

2. **Batching** (GPU, shared): Successfully prepared samples are accumulated into batches and dispatched to the GPU.  A GPU-level failure (OOM, CUDA error) is shared by all requests in the batch and mapped to HTTP 503 via `_ForwardError`.

This separation guarantees that one user's malformed payload can never cause failures for other users' valid requests.

## BatchScheduler

```python theme={null}
class BatchScheduler(model: BaseServerModel,
                     max_batch_size: int = _DEFAULT_MAX_BATCH_SIZE,
                     max_wait_ms: float = _DEFAULT_MAX_WAIT_MS,
                     preprocess_workers: int = _DEFAULT_PREPROCESS_WORKERS,
                     mode: InferenceMode = InferenceMode.PREDICTION)
```

Two-stage pipeline: parallel deserialization then batched GPU inference.

Stage 1 — Deserialization (per-request, parallel): Each [`predict`](/sdk-reference/runtime/batch_scheduler#predict) call submits the request to a thread pool for image decoding, transform, shape validation, and (for M-Optimus) bulk RNA preprocessing.  If any step fails, the request's future is immediately resolved with a `ValueError` (→ HTTP 400) and the request never enters the batch queue.

Stage 2 — Batching (shared GPU): Valid [`PreparedSample`](/sdk-reference/runtime/server_model#preparedsample) objects are collected into batches of up to *max\_batch\_size* and dispatched to the GPU.  A forward failure raises `_ForwardError` (a `RuntimeError` subclass → HTTP 503) to all requests in that batch.

<ParamField body="model">
  The server-side model to batch requests for.
</ParamField>

<ParamField body="max_batch_size">
  Maximum tiles per GPU forward pass.
</ParamField>

<ParamField body="max_wait_ms">
  Maximum milliseconds to wait before dispatching a partial batch.
</ParamField>

<ParamField body="preprocess_workers">
  Thread pool size for parallel deserialization.
</ParamField>

<ParamField body="mode">
  `"prediction"`, `"embedding"`, or `"prediction_with_embedding"`.
</ParamField>

***

#### shutdown

```python theme={null}
def shutdown() -> None
```

Releases resources owned by the scheduler.

Shuts down the preprocessing thread pool (without waiting) and cancels the batch-loop task if it is still pending. Safe to call multiple times and from outside the scheduler's event loop (e.g. when pruning a scheduler whose loop has already closed).

***

#### predict

```python theme={null}
async def predict(request: ModelRequest) -> ModelResponse
```

Deserializes, validates, and enqueues a request for batched inference.

Deserialization runs in a thread pool so the event loop is not blocked.  If deserialization fails, the error is raised directly to the caller (HTTP 400).  Otherwise the prepared sample enters the batch queue and awaits GPU execution.

<ParamField body="request">
  The tile request to process.
</ParamField>

**Returns**:

The model response for this tile.