Tile embeddings & PCA

Both H-Optimus and M-Optimus produce a 1536-dimensional embedding per tile (the CLS token). Principal components of these embeddings reveal the dominant axes of morphological variation — often corresponding to tissue vs. background, tumor vs. stroma, and immune-rich vs. immune-poor regions.

H-Optimus notebook

h1-jumpstart — a full, runnable H-Optimus embedding example.

M-Optimus notebook

m-jumpstart — the equivalent for M-Optimus.

1. Configure the model

Embeddings are available from both models. Import once, then pick your backend and model — the rest of the guide is identical.

from bioptimus.models.backbones import Backbone
from bioptimus.models.types import Models
from bioptimus.preprocess.wsi.models.bioptimus_mask import BioptimusTissueMaskModel
from bioptimus.preprocess.wsi.provider.tiled import TiledTissueMask

On-premise
AWS SageMaker

API_URL = "http://localhost:8080"
model  = Backbone(Models.H1, backend="remote", base_url=API_URL)
tissue = Backbone("tissue-seg", backend="remote", base_url=API_URL)

API_URL = "http://localhost:8080"
model  = Backbone(Models.M_OPTIMUS, backend="remote", base_url=API_URL)
tissue = Backbone("tissue-seg", backend="remote", base_url=API_URL)

model  = Backbone(Models.H1, backend="aws",
                  endpoint_name="h-optimus", region_name="us-east-1")
tissue = Backbone("tissue-seg", backend="aws",
                  endpoint_name="h-optimus", region_name="us-east-1")

model  = Backbone(Models.M_OPTIMUS, backend="aws",
                  endpoint_name="m-optimus", region_name="us-east-1")
tissue = Backbone("tissue-seg", backend="aws",
                  endpoint_name="m-optimus", region_name="us-east-1")

mask_provider = TiledTissueMask(model=BioptimusTissueMaskModel(backbone=tissue))

2. Extract embeddings

from bioptimus.inference.inference import SlideInference as Inference
from bioptimus.inference.writers import OutputFormat

inferrer = Inference(wsi_path="/data/wsi/tcga_coad.svs", model=model,
                     mask_provider=mask_provider, mask_threshold=0.5)
result_path = inferrer.predict(output_path="/data/output/tcga_coad.zarr",
                               output_format=OutputFormat.ZARR,
                               max_concurrency=256, mode="embed")

On a single-GPU SageMaker endpoint (ml.g5.xlarge), lower max_concurrency (e.g. 32) to avoid out-of-memory from concurrent tile requests.

3. PCA

Load the embeddings and fit PCA in one call — bioptimus.utils.plot_pca_scatter fits, plots, and returns the scores plus the fitted PCA (explained-variance ratios are printed):

from bioptimus import utils

data = utils.load_zarr_output(result_path)
embeddings, coords = data["outputs"], data["coords"]   # (N, 1536), (N, 2)

scores, pca = utils.plot_pca_scatter(embeddings, n_components=3,
                                     title="Embeddings PCA", cmap="plasma")

Scatter plot of the first two principal components of tile embeddings — PCA of tile embeddings on a TCGA-LUAD slide (TCGA-75-7027); PC1 explains 27.5%, PC2 13.0% of variance. Points are colored by tile index. Representative example.

4. Spatial visualization

plot_spatial_pca maps the PC scores back onto tile coordinates as per-PC spatial heatmaps — each component tends to isolate a distinct tissue compartment:

utils.plot_spatial_pca(scores, coords, components=[0, 1, 2])

Three spatial heatmaps, one per principal component, overlaid on tile coordinates — Spatial heatmaps of PC1–PC3 on a TCGA-LUAD slide (TCGA-75-7027). Each principal component highlights a different tissue compartment. Representative example.

The same embeddings feed downstream models (classification, clustering, retrieval, MIL). For the high-level workflow with caching and workspaces, use the Inference pipeline with infer.embed(path).

Overview

Get Started

Preprocessing

Workflows

Reference

H-Optimus notebook

M-Optimus notebook

1. Configure the model

2. Extract embeddings

3. PCA

4. Spatial visualization

H-Optimus notebook

M-Optimus notebook

​1. Configure the model

​2. Extract embeddings

​3. PCA

​4. Spatial visualization

1. Configure the model

2. Extract embeddings

3. PCA

4. Spatial visualization