bioptimus.observability.probes

Fail-safe environment probes for diagnostic snapshots. Captures a point-in-time picture of the runtime environment that matters for inference and reports it as a plain, JSON-friendly mapping that feeds both the support bundle (diagnostics.json) and the server’s /diagnostics endpoint. The snapshot covers:

installed SDK and dependency versions (probe_versions);
installed dependencies versus the SDK’s declared constraints (probe_dependencies);
host platform and resources (probe_host);
inference-relevant environment variables (probe_env);
GPU, CUDA, and PyTorch runtime state (probe_gpu);
importable whole-slide-image backends (probe_wsi_backends);
temporary-directory headroom and leaked AOTInductor extractions (probe_disk);
.pt2 checkpoint architecture compatibility (probe_pt2_compat); and
loaded models (probe_models).

The assembled snapshot is stamped with a schema version and a capture timestamp so it can be parsed and correlated with the logs over time. Every probe is fail-safe: it never raises. Any failure is captured in the returned data (for example {"available": false, "reason": ...} or an "error" key) so a broken environment still yields a usable snapshot.

probe_wsi_backends

def probe_wsi_backends() -> dict[str, dict[str, Any]]

Probes whether each supported WSI backend can be imported. Attempts to import each backend library, recording its version when the import succeeds and the failure reason (for example a missing native library) when it does not. Returns: Mapping of backend name to a status dict: available is True with a version when the backend imports, otherwise False with a reason.

probe_disk

def probe_disk() -> dict[str, Any]

Probes free space and AOTInductor temp-dir usage in the system tempdir. Reports the temporary directory used for .pt2 extraction, its free space and percentage used, whether it is writable, and how many leftover AOTInductor extraction directories it contains (a count that grows when packages are not cleaned up). Sets warning to "TMP_NOT_WRITABLE" when the directory cannot be written (which breaks .pt2 extraction) or "TMP_NEARLY_FULL" when usage is high. Returns: A status dict with tmp_path, tmp_free_gb, tmp_pct_used, aotinductor_temp_dirs, writable, and warning (None when healthy), or {"error": ...} if the directory cannot be inspected.

probe_models

def probe_models(
        models: Mapping[str, object | None] | None = None
) -> list[dict[str, Any]]

Summarizes the supplied models for the diagnostic snapshot. Accepts a mapping of model name to the loaded model object, or None for a model that is configured but failed to load. Each model is reported with its load state and, when loaded, its device and precision. Reading a single model never breaks the probe.

Mapping of model name to loaded model object (or None when it failed to load). Omitted or empty yields an empty list.

Returns: A list of per-model status dicts, each with name and loaded and, for loaded models, device and precision.

probe_versions

def probe_versions() -> dict[str, str | None]

Probes installed versions of the SDK and its key dependencies. Reads versions from installed package metadata (not module attributes), so it works even when a package cannot be imported. Dependency drift and ABI mismatches (for example a torch that does not match the compiled .pt2 artifacts) are a common, hard-to-spot cause of failures. Returns: Mapping of distribution package name to its installed version string, or None when the package is not installed.

probe_dependencies

def probe_dependencies() -> dict[str, Any]

Checks installed dependencies against the SDK’s declared constraints. Reads the version constraints the installed bioptimus-sdk distribution declares for its dependencies and verifies that each installed package satisfies them. Where probe_versions reports the raw installed numbers, this probe reports the verdict: upgrading a dependency past a supported ceiling (for example zarr<3.0.0 or torch<2.10.0, which the .pt2 artifacts are compiled against) is a common, silent cause of breakage that a bare version list does not reveal. Returns: A status dict with distribution, satisfied (True when every applicable constraint holds), conflicts (a {"name", "installed", "required"} entry per violation), and checked (the number of constraints evaluated), or {"error": ...} when the constraints cannot be evaluated.

probe_host

def probe_host() -> dict[str, Any]

Probes host platform and resource information. Reports the Python and OS platform, the interpreter running the SDK (a common “installed it but it is not found” culprit when it is the wrong one), CPU count and affinity, total physical RAM, any container memory limit, the open-file-descriptor limit (which concurrent tile reads can exhaust), and system load average. Each resource field degrades to None when it is unavailable on the host (for example load average on non-Unix platforms, or a container limit outside a container). Returns: A mapping with python_version, platform, machine, executable, in_venv, cpu_count, cpu_affinity, total_ram_gb, container_memory_limit_gb, open_file_limit, and load_avg.

probe_env

def probe_env() -> dict[str, str | None]

Probes a fixed allowlist of inference-relevant environment variables. Only the allowlisted, non-secret names in _TRACKED_ENV_VARS are read; the full environment is never captured, so credentials and tokens are never exposed. Returns: Mapping of environment-variable name to its value, or None when the variable is unset.

probe_gpu

def probe_gpu() -> dict[str, Any]

Probes GPU, CUDA, and PyTorch runtime state. Reports the PyTorch and CUDA build versions, the NVIDIA driver version, the CUDA architectures PyTorch was compiled for, and whether CUDA is available, plus per-device properties (name, compute capability, whether PyTorch’s compiled kernels cover it, total and free memory). The driver version is captured even when CUDA is unavailable, since a driver too old for the CUDA build is a leading cause of cuda_available being false. A device whose compute capability is missing from PyTorch’s build (for example after a wheel reinstall) loads but then fails at kernel launch, so it is flagged here. When CUDA is available it also verifies the context is responsive with a small device-to-host copy, since a present GPU with a corrupted context still fails inference. Returns: A mapping describing the GPU and runtime state. It always includes cuda_available, and includes reason when PyTorch is missing or CUDA cannot be queried.

probe_pt2_compat

def probe_pt2_compat(
    checkpoints: Mapping[str, str | Path] | None = None
) -> list[dict[str, Any]]

Checks exported .pt2 checkpoints against the host architecture. The AOTInductor .pt2 artifacts are compiled for a specific GPU (and CPU) architecture; running an artifact compiled for a different architecture fails at load or execution time. This probe reports, per checkpoint, the architecture it was compiled for and whether it matches the current host.

Mapping of model name to .pt2 checkpoint path. Omitted or empty yields an empty list.

Returns: A list of per-checkpoint status dicts, each with name, checkpoint, compiled_for, cpu_arch, compatible, and reason (the mismatch message, or None when compatible).

snapshot

def snapshot(
        models: Mapping[str, object | None] | None = None,
        checkpoints: Mapping[str, str | Path] | None = None) -> dict[str, Any]

Assembles a full diagnostic snapshot of the runtime environment. Combines every probe into one JSON-friendly mapping. Never raises: each section is captured independently, so a failing probe degrades to an {"error": ...} section instead of breaking the snapshot.

Optional mapping of model name to loaded model object (or None when it failed to load), forwarded to probe_models.

Optional mapping of model name to .pt2 checkpoint path, forwarded to probe_pt2_compat.

Returns: A mapping stamped with schema_version and captured_at (UTC ISO-8601), plus the versions, dependencies, host, env, gpu, wsi_backends, disk, pt2_compat, and models sections.

Overview

Get Started

Preprocessing

Workflows

Troubleshooting

Reference

probe_wsi_backends

probe_disk

probe_models

probe_versions

probe_dependencies

probe_host

probe_env

probe_gpu

probe_pt2_compat

snapshot

​probe_wsi_backends

​probe_disk

​probe_models

​probe_versions

​probe_dependencies

​probe_host

​probe_env

​probe_gpu

​probe_pt2_compat

​snapshot

probe_wsi_backends

probe_disk

probe_models

probe_versions

probe_dependencies

probe_host

probe_env

probe_gpu

probe_pt2_compat

snapshot