ODT_Reconstruction

Automated pipeline for Optical Diffraction Tomography (ODT) — reconstructing 3D refractive index maps of biological samples from interferometric holographic measurements on an HPC cluster.

First time here? — One-time setup (4 steps, do once)

These 4 steps are needed only once per cluster account. After that, use the 2 steps above.

1. Log in to the cluster

Open a terminal (on Windows: Command Prompt or PowerShell) and connect:

ssh <your_username>@zpe.intranet.mpl.mpg.de

2. Get the code

cd ~
git clone https://github.com/OpticalDiffractionTomography/ODT_Reconstruction.git
cd ODT_Reconstruction

3. Set your email and check the paths

Open the config file:

nano config.sh

Set your email and check that the two mount paths are correct for your cluster:

EMAIL="you@institute.de"                    # ← set your email here
DATA_MOUNT="/mnt/guck_division2"            # ← where your raw data is
RESULTS_MOUNT="/mnt/ZPE_cluster_results"    # ← where results are written

Save and close: press Ctrl+O, then Enter, then Ctrl+X.

Everything else in the file has working defaults — leave it as is.

4. Install the command

bash install.sh
source ~/.bashrc

Check that it works:

tomo_process --help

That's it — from now on you only need the 2 steps in Run jobs below.

---

Done with setup? — Run jobs (2 steps)

Step 1 — log in to the cluster:

ssh <your_username>@zpe.intranet.mpl.mpg.de

Step 2 — start processing:

tomo_process --path "<paste your dataset path here>"

Example — copy the folder path from Windows File Explorer and paste it as it is:

tomo_process --path "U:\Members\YourName\20260715_experiment"

Done — you can close the terminal immediately; processing continues on the cluster. You will get an email when it starts and when it finishes.

Good to know:

• Windows paths (drive letter, backslashes) are converted automatically — no need to edit them.
• The folder can contain multiple subdirectories; all sample*_Tomog.mat files are found and processed.
• Re-running the same dataset overwrites the old results.
• Need to change the refractive index of the medium? Add --nm, e.g. --nm 1.340 (default: 1.337). See Options.

Where results appear: on the results mount, following the same folder structure as your data. For the example above:

Your data : <DATA_MOUNT>/Members/YourName/20260715_experiment/
Results   : <RESULTS_MOUNT>/Members/YourName/20260715_experiment/field_retrieval_zpe_results/

If your experiment folder contains subfolders (e.g. one per batch), each subfolder gets its own field_retrieval_zpe_results/ next to its sample files.

(DATA_MOUNT and RESULTS_MOUNT are the two paths set in config.sh during setup.)

---

Options

Option	What it does	Default
--nm 1.340	Refractive index of the culture medium	1.337
--email you@lab.de	Notification email for this run only	EMAIL from config.sh
--max-jobs 3	Parallel SLURM jobs	5
--mins-per-sample 10	Time estimate per sample (used to size jobs)	15
-v alice / --version alice	Alice's variant: only the first bg*_Tomog.mat per folder is used (outputs have no _bg_<N> suffix), and TIFFs are written in a final export step as <batchName><NNN>_Tomo.tif. Everything else is identical to default. See Backgrounds and pipeline versions.	default

Run tomo_process --help for the full list.

Example commands

Copy the line that matches what you want, then replace the folder path with your own. Options can be combined in any order.

Normal run (most people only need this):

tomo_process --path "U:\Members\YourName\20260715_experiment"

My medium has a different refractive index (e.g. 1.340):

tomo_process --path "U:\Members\YourName\20260715_experiment" --nm 1.340

Send the email to a different address (just for this run):

tomo_process --path "U:\Members\YourName\20260715_experiment" --email you@lab.de

Use Alice's variant (only the first background per folder is used):

tomo_process --path "U:\Members\YourName\20260715_experiment" -v alice

Everything together:

tomo_process --path "U:\Members\YourName\20260715_experiment" --nm 1.340 --email you@lab.de -v alice

Large dataset — limit how many cluster jobs run at once (here 3):

tomo_process --path "U:\Members\YourName\20260715_experiment" --max-jobs 3

Samples take longer than usual (here ~30 minutes each, so jobs are split smaller):

tomo_process --path "U:\Members\YourName\20260715_experiment" --mins-per-sample 30

---

Monitoring a run

tomo_process --list                          # list all runs and their status
tomo_process --status tomo_20260729_143201   # detailed status of one run
tomo_process --cancel tomo_20260729_143201   # cancel a run and its active jobs

Reading the status output

=== Run: tomo_20260729_143201 ===
  Total chunks : 6      ← total number of jobs submitted
  Done         : 4      ← completed and results copied back
  Active       : 2      ← currently running on cluster
  Pending      : 0      ← waiting to be submitted
  Failed       : 0      ← check logs if this is non-zero

Full orchestrator log:

$SCRATCH_ROOT/<run_id>/logs/orchestrator.log

Per-job MATLAB output:

$SCRATCH_ROOT/<run_id>/logs/tomo_<run_id>_chunk000N.out

---

If your SSH disconnects or the login node reboots

A dropped SSH connection does not interrupt processing. If the login node itself reboots, no work is lost — just reconnect and resume:

ssh <your_username>@zpe.intranet.mpl.mpg.de
tomo_process --list                          # find your run_id
tomo_process --resume tomo_20260729_143201   # continue from where it stopped

---

Updating to the latest version

tomo_process --upgrade

This pulls the newest code and keeps your local edits (such as the email in config.sh). If the update ever conflicts with your edits, your changes stay safe in git stash list and the command tells you so. Runs already in progress keep using the old code until you run tomo_process --restart <run_id>.

---

Data requirements

The pipeline accepts two kinds of folder layout — use whichever matches how your experiment was recorded.

1. A single experiment folder — one background file plus its sample files, all directly inside the folder you pass to --path:

20260715_experiment/
  bg001_Tomog.mat
  sample001_Tomog.mat
  sample002_Tomog.mat
  sample003_Tomog.mat

2. An experiment split into batches — a batch01, batch02, ... subfolder for each batch, each with its own background and sample files:

20260715_experiment/
  batch01/
    bg001_Tomog.mat
    sample001_Tomog.mat
    sample002_Tomog.mat
  batch02/
    bg001_Tomog.mat
    sample001_Tomog.mat
    sample002_Tomog.mat

In this case, each batch is processed on its own — the samples in batch01 are only ever combined with the background from batch01, never with batch02's background.

File names can vary a little (e.g. bg_Tomog.mat, bg001_Tomog.mat, or sample001_TimeLapse_001_Tomog.mat) — any file name starting with bg or sample and ending in Tomog.mat is recognized.

Backgrounds and pipeline versions (--version)

How backgrounds are paired with samples, and how the TIFFs are named, depends on --version (-v). Everything else (phase retrieval, outlier-frame detection, reconstruction, .mat/.png outputs) is identical in both versions.

	default	alice (-v alice)
Backgrounds used	Every bg*Tomog.mat in the folder	Only the first bg*Tomog.mat in the folder (alphabetical order); the others are ignored and listed in the log
Samples processed	Once per background	Once
_bg_<N> suffix on outputs	Yes	No
When TIFFs are written	Right after each sample is reconstructed	In one final export step, after all samples in the folder are reconstructed
TIFF name	Tomogram_Field_<sample>_bg_<N>.tif	<folderName><NNN>_Tomo.tif

Example — a folder with two backgrounds and two samples:

batch01/
  bg001_Tomog.mat
  bg002_Tomog.mat
  sample001_Tomog.mat
  sample002_Tomog.mat

default                                alice
───────                                ─────
bg001 → sample001, sample002           bg001 → sample001, sample002
bg002 → sample001, sample002           (bg002 ignored)
= 4 reconstructions                    = 2 reconstructions

Notes:

• Batches stay separate in both versions. A sample is only ever combined with a background from its own folder.
• Frame matching. Each sample frame is divided by the frame with the same index in the background stack, in both versions.
• "First" background. This is the first in MATLAB's alphabetical (case-sensitive) order, so bg001_Tomog.mat comes before bg_Tomog.mat. If the folder has only one background, the two versions pair samples identically and differ only in file names.
• Alice TIFF numbering. <folderName> is the name of the folder that holds the sample files. <NNN> is the sample's position (001, 002, ...) in alphabetical order within that folder. A large folder can be split over several cluster jobs; the numbering continues across those jobs, so TIFFs never overwrite each other.
• TIFF format is the same in both versions: multi-page uint16, RI × 10000, each slice mirrored left–right.

What the pipeline produces

For each subdirectory, results are written to $RESULTS_MOUNT/<rel_path>/field_retrieval_zpe_results/.

default (one set per sample and background, _bg_<N> = background number):

field_retrieval_zpe_results/
  Field_sample001_Tomog_bg_1.mat            # retPhase, retAmplitude, NA, lambda, res, ZP, f_dx, f_dy
  Field_sample001_Tomog_bg_1.png            # diagnostic phase overview image
  Tomogram_Field_sample001_Tomog_bg_1.mat   # Reconimg (3D RI volume), res3, res4, lambda, excludeFrame
  Tomogram_Field_sample001_Tomog_bg_1.tif   # multi-page uint16 TIFF (values × 10000)
  Tomogram_Field_sample001_Tomog_bg_1.png   # diagnostic orthogonal slice image
  Field_inspection_chunk0000.png            # mean phase and frame-diff curves across all samples
  field_retrieval_chunk0000.log             # Stage 1 log
  tomogram_reconstruction_chunk0000.log     # Stage 2 log

alice (one set per sample, no _bg_<N>; TIFFs named after the folder):

field_retrieval_zpe_results/
  Field_sample001_Tomog.mat
  Field_sample001_Tomog.png
  Tomogram_Field_sample001_Tomog.mat
  Tomogram_Field_sample001_Tomog.png
  batch01001_Tomo.tif                       # <folderName><NNN>_Tomo.tif
  batch01002_Tomo.tif
  Field_inspection_chunk0000.png
  field_retrieval_chunk0000.log
  tomogram_reconstruction_chunk0000.log

The _chunkNNNN suffix on logs and the inspection plot identifies the cluster job that produced them. When a folder is split over several jobs, each job writes its own copy.

---

Advanced

How job chunking works

Jobs are auto-sized from the walltime and the per-sample time estimate:

max_samples_per_job = floor(SLURM_TIME / MINS_PER_SAMPLE)
                    = floor(20h × 60min / 15min)
                    = 80 samples per job

Files from different subdirectories are never mixed in one job — each subdirectory is processed independently, split across multiple jobs if it contains more samples than the per-job limit.

How it works internally

tomo_process (runs on login node, exits immediately after submitting)
  │
  └─► nohup scripts/orchestrator.sh (stays running on login node as background process)
            │
            │  Groups sample files by subdirectory, auto-sizes chunks from walltime
            │
            ├─ [login node] rsync chunk files: DATA_MOUNT → SCRATCH_ROOT/<run>/<chunk>/
            ├─ sbatch job script (up to 5 jobs in parallel on GPU nodes)
            │      └─ field_Retrieval.m          Stage 1: complex field retrieval
            │      └─ tomogram_Reconstruction.m  Stage 2: 3D RI reconstruction
            │      └─ touch DONE or FAIL marker
            │
            ├─ [login node] rsync results: SCRATCH_ROOT/<run>/<chunk>/ → RESULTS_MOUNT
            ├─ rm scratch data for that chunk
            └─ repeat until all chunks done, then send notification email

The login node handles all network mount access (DATA_MOUNT / RESULTS_MOUNT) because compute nodes typically do not have those mounts. Only scratch (SCRATCH_ROOT) needs to be accessible from compute nodes.

If a pasted --path cannot be found under DATA_MOUNT, the script strips the drive letter and drops up to 3 leading folders while searching; if it still fails, it lists the paths it tried and how to fix yours.

Manual submission (single dataset, advanced)

If your dataset fits within scratch space and you want to submit directly without the orchestrator:

sbatch scripts/main.sh --data_dir /path/to/experiment_data
sbatch scripts/main.sh --data_dir /path/to/experiment_data -v alice   # Alice's variant

--data_dir may be a single experiment folder or one containing batch*/ subfolders. Results go to field_retrieval/ inside each folder, and logs and the inspection plot have no _chunkNNNN suffix. In alice mode the TIFF numbering starts at 001 in each folder.

scripts/main.sh runs both stages sequentially on one GPU node. It does not handle copying from/to the network mounts — you must do that manually.

Pipeline stages (technical reference)

Stage 1 — Field Retrieval (src/field_Retrieval.m)

For each sample*_Tomog.mat file:

1. Detect off-axis carrier frequency from background frames via FFT peak
2. Build circular demodulation mask (mk_ellipse)
3. Demodulate each frame: shift +1 diffraction order to centre, divide by background
4. Phase pipeline per frame: angle → unwrap2 → PhiShift → phaseCompensation(deg=1) → cell-exclusion mask → phaseCompensation(deg=2)
5. Save retPhase, retAmplitude and diagnostic PNG

Frames where unwrap2 fails (e.g. saturated or empty patches) are set to NaN and excluded automatically in Stage 2.

Stage 2 — Tomogram Reconstruction (src/tomogram_Reconstruction.m)

For each Field_*.mat output from Stage 1:

1. Detect outlier frames (excludeFrame): mean phase > 1.5, frame-to-frame diff > 0.1, NaN frames
2. Build TomoParam struct with all optical/geometric parameters
3. ODTReconstruction — Ewald sphere mapping into 3D Fourier volume
4. ODTIteration (100 iterations, GPU) — enforce n ≥ n_medium physical constraint
5. Save Reconimg as .mat, multi-page .tif, and diagnostic .png

Key physical parameters

Variable	Meaning
lambda	Illumination wavelength (µm)
NA	Objective numerical aperture
res	Camera pixel size (µm)
n_m	Refractive index of culture medium (default 1.337; override with --nm)
n_s	Expected sample RI (cells); always n_m + 0.04
ZP	Hologram FFT size
ZP2 = 512	Lateral size of 3D Fourier volume
ZP3 = 256	Axial size of 3D Fourier volume

Stay connected during setup (tmux)

If your SSH session drops during the one-time setup, interactive commands die with it. Start a persistent shell first:

tmux new -s setup

Once tomo_process is running, this is not needed — it detaches automatically.

Tests

bash tests/run_tests.sh

Shell tests need only bash (squeue, rsync, sbatch and matlab are mocked):

File	Covers
tests/test_orchestrator.sh	Failure handling in scripts/orchestrator.sh: retry/bisection, salvage copy, lost-job detection, TIFF index
tests/test_submit_chunk.sh	Staging and submission: background/sample copy, skipped files, job-script filling, TIFF offset/index
tests/test_job_template.sh	The real scripts/bash_template.sh against a fake matlab: DONE/FAIL markers, Stage 1 crash handling, SIGTERM
tests/test_tomo_process.sh	tomo_process --restart/--resume/--cancel/--upgrade and argument handling

MATLAB unit tests run automatically if matlab is on PATH, or manually with runtests('tests') from the repo root:

File	Covers
tests/tLoadMatSafe.m	loadMatSafe, recordSkipped (corrupt, truncated and incomplete MAT files)
tests/tHelpers.m	mk_ellipse, PhiShift, phaseCompensation, saveTomogramTIFF, saveTomogramPNG, saveInspectionPNG