Spatial transcriptomics

M-Optimus predicts expression of thousands of genes directly from H&E histology — no spatial transcriptomics assay required — enabling spatial analysis on any archived slide. This guide uses the core API (Backbone + SlideInference) for explicit control; the Inference pipeline wraps the same steps.

Runnable notebook

See m-jumpstart for a full, runnable M-Optimus example.

1. Configure the model and tissue masking

Pick your backend — the rest of the guide is identical for both.

On-premise
AWS SageMaker

from bioptimus.models.backbones import Backbone
from bioptimus.models.types import Models
from bioptimus.preprocess.wsi.models.bioptimus_mask import BioptimusTissueMaskModel
from bioptimus.preprocess.wsi.provider.tiled import TiledTissueMask

API_URL = "http://localhost:8080"
model = Backbone(Models.M_OPTIMUS, backend="remote", base_url=API_URL)
tissue_backbone = Backbone("tissue-seg", backend="remote", base_url=API_URL)

from bioptimus.models.backbones import Backbone
from bioptimus.models.types import Models
from bioptimus.preprocess.wsi.models.bioptimus_mask import BioptimusTissueMaskModel
from bioptimus.preprocess.wsi.provider.tiled import TiledTissueMask

model = Backbone(Models.M_OPTIMUS, backend="aws",
                 endpoint_name="m-optimus", region_name="us-east-1")
tissue_backbone = Backbone("tissue-seg", backend="aws",
                           endpoint_name="m-optimus", region_name="us-east-1")

mask_provider = TiledTissueMask(model=BioptimusTissueMaskModel(backbone=tissue_backbone))
print(f"{len(model.input_gene_names or [])} input genes, "
      f"{len(model.output_gene_names or [])} output genes")

2. Run inference (image only)

SlideInference tiles the slide at 0.5 µm/pixel, dispatches tiles concurrently, and streams results to Zarr so memory stays constant regardless of slide size.

from bioptimus.inference.inference import SlideInference
from bioptimus.inference.writers import OutputFormat

inferrer = SlideInference(
    wsi_path="/data/wsi/tcga_coad.svs",
    model=model,
    mask_provider=mask_provider,
    mask_threshold=0.5,
)
result_path = inferrer.predict(
    output_path="/data/output/tcga_coad.zarr",
    output_format=OutputFormat.ZARR,
    max_concurrency=256,                # see note for single-GPU SageMaker endpoints
)

Tissue masking skips background tiles before prediction (typically most of a slide’s tile grid). mask_threshold sets the minimum tissue fraction to keep a tile (default 0.5).

On a single-GPU SageMaker endpoint (ml.g5.xlarge), lower max_concurrency (e.g. 32) to avoid out-of-memory from concurrent tile requests. On a multi-GPU on-premise server, 256 is a reasonable default. See Performance tuning for more.

3. Add bulk RNA (multimodal prediction)

M-Optimus prediction can be conditioned on bulk RNA for a multimodal readout, attached through a Cohort via late-binding: run image-only first, then link RNA and re-predict. Modality-aware tracking reruns only what changed, keeping both outputs for comparison.

Send TPM-normalized expression values — do not log1p them yourself. The server transforms the input and the model output for you. Genes are auto-aligned to the model’s input set; genes the model expects but your file lacks are encoded as -1 (non-measured).

The server applies two transforms around the model:

Stage	Transform	Applied to
Input (preprocess)	`log1p`	your TPM-normalized expression values, before the model
Output (postprocess)	`expm1`	the model output, to return expression values

from bioptimus.data.cohort import Cohort
from bioptimus.inference import Inference
from bioptimus.models.types import Models

cohort = Cohort.from_directories(wsi_dir="/data/wsi/tcga_mini_coad")
infer_m = Inference(model_name=Models.M_OPTIMUS, cohort=cohort,
                    api_url="http://localhost:8080", tissue=True,
                    output_path="/data/output", experiment="tcga_coad", run=1)
infer_m.run(mode="predict")            # image-only baseline

Link bulk RNA and re-predict:

# Plain CSV (Ensembl IDs as columns):
cohort.link_bulk_rna("/data/rna/tcga_mini_coad")

# GDC/TCGA gene-quantification TSVs — use the tpm_unstranded column (not
# fpkm_* or raw counts) and strip Ensembl versions:
cohort.link_bulk_rna(
    "/data/rna/tcga_mini_coad",
    gene_column="gene_id",
    value_column="tpm_unstranded",      # TPM-normalized expression values (the expected scale)
    strip_version=True,                 # ENSG00000000003.15 -> ENSG00000000003
)

infer_m.run(mode="predict", force=True)  # re-predict with bulk RNA context

Retrieve each stage’s output by modality:

out = cohort[0].outputs.get(infer_m.model_name)
img_only   = out.get_stage_path("predict", ["image"])
multimodal = out.get_stage_path("predict", ["image", "bulk_rna"])

4. Inspect the output

Load either output — image-only or multimodal — with the built-in helper:

from bioptimus import utils

data = utils.load_zarr_output(result_path)
outputs, coords = data["outputs"], data["coords"]   # (n_tiles, n_genes), (n_tiles, 2)
gene_names = data["gene_names"]

5. Visualize

Genes are identified by Ensembl gene ID. This guide’s panel:

Gene	Ensembl ID	Marker for
EPCAM	`ENSG00000119888`	Epithelium / tumor
CD3E	`ENSG00000198851`	T-cells
CD8A	`ENSG00000153563`	Cytotoxic T-cells
MKI67	`ENSG00000148773`	Proliferation
COL1A1	`ENSG00000108821`	Stroma
SFTPC	`ENSG00000168484`	Alveolar (lung)

bioptimus.utils.plot_gene_panel_overlay renders a whole panel in one call. Compare the step 3 outputs — multimodal (with bulk RNA) vs. image-only:

from bioptimus import utils

GENE_PANEL = {
    "EPCAM": "ENSG00000119888",
    "CD3E": "ENSG00000198851",
    "CD8A": "ENSG00000153563",
    "MKI67": "ENSG00000148773",
    "COL1A1": "ENSG00000108821",
    "SFTPC": "ENSG00000168484",
}

With bulk RNA (multimodal)
Without bulk RNA (image only)

bulk = utils.load_zarr_output(multimodal)
utils.plot_gene_panel_overlay(bulk["outputs"], bulk["coords"], bulk["gene_names"],
                              GENE_PANEL, wsi_path=wsi_path, cols=3, cmap="inferno")

Six-gene spatial expression panel, image plus bulk RNA — Panel predicted from H&E with bulk RNA (TCGA-LUAD TCGA-75-7027) — the multimodal output. Representative example.

image = utils.load_zarr_output(img_only)
utils.plot_gene_panel_overlay(image["outputs"], image["coords"], image["gene_names"],
                              GENE_PANEL, wsi_path=wsi_path, cols=3, cmap="inferno")

Six-gene spatial expression panel, image only — The same panel from H&E only, with no bulk RNA (TCGA-LUAD TCGA-75-7027). Comparing the tabs shows how bulk RNA refines the prediction without retraining. Representative example.

A FP16 model variant is available for faster inference. Run the FP32 and FP16 outputs through the same analysis to compare the impact of quantization.

6. Get embedding + prediction together (optional)

Need both outputs for the same tiles? Request the combined mode instead of calling predict() and embed() separately — one backbone pass instead of two:

combined_path = inferrer.predict(
    output_path="/data/output/tcga_coad_combined.zarr",
    output_format=OutputFormat.ZARR,
    max_concurrency=256,
    mode="prediction_with_embedding",
)

The output file carries both arrays, keyed by the same tile coordinates:

Array	Contents
`outputs`	Gene prediction (same as step 2)
`embeddings`	1536-d tile embedding

Reduce each to a slide-level vector:

from bioptimus.inference.aggregate import pseudobulk, slide_embedding

expression = pseudobulk(combined_path).expression   # from `outputs`
embedding = slide_embedding(combined_path)          # from `embeddings`

M-Optimus only — raises NotImplementedError on the H-Optimus family (embedding-only, no prediction head), on any backend including Hugging Face. Call embed() directly for those. On M-Optimus itself, call predict()/embed() separately if you want both outputs without the combined call.

Overview

Get Started

Preprocessing

Workflows

Troubleshooting

Reference

Runnable notebook

1. Configure the model and tissue masking

2. Run inference (image only)

3. Add bulk RNA (multimodal prediction)

4. Inspect the output

5. Visualize

6. Get embedding + prediction together (optional)

Runnable notebook

​1. Configure the model and tissue masking

​2. Run inference (image only)

​3. Add bulk RNA (multimodal prediction)

​4. Inspect the output

​5. Visualize

​6. Get embedding + prediction together (optional)

1. Configure the model and tissue masking

2. Run inference (image only)

3. Add bulk RNA (multimodal prediction)

4. Inspect the output

5. Visualize

6. Get embedding + prediction together (optional)