daoTools

daoTools is the application layer of DAO (Durham Adaptive Optics), a real-time control system for adaptive optics instruments. It builds on daoBase's shared-memory (dao.shm) and messaging primitives to provide everything needed to run an AO real-time control loop end to end: pixel calibration, wavefront-sensor centroiding (Shack-Hartmann, pyramid, correlation-based), matrix-vector-multiply reconstruction (CPU/BLAS and GPU/CUDA), leaky-integrator and modal loop control, plus the calibration, simulation, timing/latency, and data-conversion tools needed to operate and debug a pipeline.

It ships three things: a C function library (libdaoTools) that the real-time per-pixel/per-mode math lives in, 70+ command-line applications in apps/ that are thin real-time wrappers around it, and a set of PyQt5 GUIs in gui/ for live monitoring and control. Everything communicates over dao.shm, so any of these pieces can be mixed, replaced, or driven from your own code independently.

Documentation

Full installation, usage, and API documentation is available on Read the Docs.

Citation

If you use DAO or daoTools in research or other published work, please cite daoBase, the core DAO library, using its DOI:

DOI: 10.5281/zenodo.17264152

@software{barr_2025_17264152,
  author       = {Barr, David and
                  Cetre, Sylvain and
                  Connolly, John and
                  Thomas Davies},
  title        = {Durham-Adaptive-Optics/daoBase: Initial Release},
  month        = oct,
  year         = 2025,
  publisher    = {Zenodo},
  version      = {v0.0.1},
  doi          = {10.5281/zenodo.17264152},
  url          = {https://doi.org/10.5281/zenodo.17264152},
}

Standard citation format:

Barr, D., Cetre, S., Connolly, J., & Thomas Davies. (2025). Durham-Adaptive-Optics/daoBase: Initial Release (v0.0.1). Zenodo. https://doi.org/10.5281/zenodo.17264152

The daoTools library

libdaoTools (built from src/c/daoTools.c, declared in include/daoTools.h) is the C function library most of the apps/ binaries below are thin command-line wrappers around. It covers the operations that recur across an AO real-time pipeline, operating directly on dao.shm IMAGEs:

• Pixel calibration — daoToolsShmCalibrate/64 (dark-subtract + flat-field, float and double precision), daoToolsShmCalibratePws (pyramid WFS variant).

• Centroiding — daoCentroidSpots (windowed center-of-gravity), daoCentroidSpotsRelative/RelativeRef (differential centroiding; daoCentroidSpotsRelativeFast: the same results bit for bit, about 3.4x faster), daoCentroidSpotsCorrelation (windowed correlation search) and, when FFTW is available, its FFT-based counterpart (src/c/daoToolsCorrFFT.c, include/daoToolsCorrFFT.h) which searches the whole sub-aperture via one FFT round trip instead of a shift search; daoCentroidPws for pyramid WFS.

• Control loop primitives — daoToolsLeakyIntegrator/Double and the per-mode daoToolsLeakyModalIntegrator/Double, daoToolsHighPassFilter/ Double, daoToolsCommandFilter; OMGI (optimized modal gain integrator): daoToolsPseudoOpenLoop, daoToolsModalGainCost/Fit.

• Pyramid WFS pixel pipelines — daoToolsShmExtract, daoToolsShmSubstractExtract and its normalized variants (...Norm, ...NormA, ...DualNorm, plus their ...Finalize companions).

• Misc. — daoDmCombine (multi-channel DM command summing), daoDescrambleOcam2Image (OCAM2K scrambled-readout reordering), daoToolsImgNormalize, daoComputeChecksum, daoRtSetup/daoToolsEnableFTZ (real-time process setup, denormal flush-to-zero), daoToolsSetRtPriority (deprecated: daoThreadSetRtPriority from daoBase's daoThread.h -- see Real-time scheduling priority below), and daoLogToFile (throttled, size-rotated file logging).

• Tool helpers — daoToolsArgName (bounded copy of a SHM name given on the command line), daoToolsShmOpen (daoShmOpen exiting with an error when the SHM cannot be opened), daoToolsInsertShmNamePrefixN (derived SHM names with a buffer size). Name buffers are DAO_SHM_NAME_LEN bytes (dao.h).

• CPU stage plugins — daoCpuRegisterStage / daoCpuFindStage (include/daoCpuStages.h): stage types written outside daoTools, run by daoCpuPipeline (see CPU pipeline below).

Most apps/*.c binaries call straight into this library, add SHM attach/argument-parsing/real-time-loop boilerplate, and nothing else — the library is where the actual per-pixel/per-mode math lives.

libdaoToolsGpu (include/daoGpuStages.h, src/gpu/daoGpuStages.cu, built when CUDA is found) holds GPU versions of the real-time steps above — pixel calibration and extraction, the Shack-Hartmann centroiders, matrix-vector multiply, gain — used by daoGpuPipeline to run a whole chain on one GPU as one CUDA graph per frame. See GPU pipeline below.

daoTools.py

src/python/daoTools.py is a general-purpose Python AO toolbox, independent of the C library above: a Fifo circular buffer, basic FITS read/write helpers, pupil/circular-mask generation, center-of-gravity (cog), a Hadamard basis, PSD/PSF computation (computePSD, computePsf, computePsfRef), a ShackHartmannWFS simulation class and a pyramid-WFS image simulator (pwfsImage). It predates and is separate from the daoToolsLib/pipeline-scaffolding modules also under src/python/.

Tools

apps/ ships 70+ command-line tools. Full usage/flags are in the Sphinx docs (waf build_docs, or docs/source/apps.rst); this is the index:

Category	Tools
Wavefront sensing & centroiding	daoComputeCentroid (core Shack-Hartmann COG) and its Pws / Relative / RelativeRef / Correlation / CorrelationFFT variants, daoComputeCentroids.py / daoComputeCentroidsSlow.py (Python reference), daoComputeIntensityPws, daoPrepCentroidLut.py, daoPrepPwfs.py
Matrix-vector multiply	daoMvM (CPU, needs BLAS), daoMvMGPU (CUDA; uses GPU SHMs in place), daoMvM.py (Python reference)
Pipelines	daoGpuPipeline (a chain of GPU stages in one process, one CUDA graph per frame, or one process per stage with -s), daoCpuPipeline (the same configuration on the CPU, one thread, the libdaoTools functions of the apps, optional workers and plugins)
Pixel calibration	daoPixelCalibrate / daoPixelCalibratePws, daoTakeBg.py
SHM utilities	daoShmMonitoring / daoShmMonitoring1Value, daoShm2Fits, daoFits2Shm.py, daoSnapshot.py, daoShmRate.py, daoShmSlice (copy a range of one SHM into another each frame)
SHM arithmetic	daoShmAdd, daoShmCombiner, daoShmConcatenate / daoShmConcatenateFine, daoAvgShm / daoAvgDoubleShm, daoStatShm, daoDownsample, daoPixelExtract, daoApplyGain
Loop & filter	daoLeakyIntegrator / daoLeakyIntegratorMap, daoPseudoOpenLoop + daoModalGainOptimizer (OMGI: the modal integrator's gains fitted while the loop runs), daoFastAndFurious.py (Fast & Furious focal-plane wavefront sensing), daoClock, daoHighPassFilter, daoModesCutoff / daoModesCutoffFull, daoCommandFilter
Timing & latency	daoTimeDiff, daoTimeDiffNCurse, daoTimeDiffStat, daoSetLatency, daoPlotLatency.py
Data conversion & I/O	daoNpy2Shm.py, daoTakeDataCubeFITS.py, daoTakeDataCubeNPY.py, daoDescrambleOcam2, daoDMSend, daoRandImageU16Write, daoRandWriter / daoRandWriterSync
Simulation	daoReconstructor.py (Python MVM reference), daoTurbulenceSimulator.py, daoPwfsSimulator.py, daoNoisyPsfGenerator.py
Logging & communication	daoRecvLogs.py, daoSendLogs.py, daoLogToScreen.py, daoProxyLog.py, daoSendCommand.py, daoStrCmd.py / daoReadStr.py / daoWriteStr.py, daoRemoteShmFileServer.py, daoRedisCheck.py, daoDAQCli.py
Real-time display (RTD)	daoImageRTD.py / daoImageRTDFloat.py, daoBarRTD.py, daoShRTD.py, daoWavefrontRTD.py, daoPlotRTD.py

GUI

gui/ (PyQt5 + pyqtgraph, source in docs/source/gui.rst) ships standalone viewer/control applications, each reading its target straight from a dao.shm stream — pass the SHM name(s) on the command line, no config file needed:

Category	Tools
Image / SHM viewing	daoShmViewer (flagship multi-panel viewer, dark by default (--light for light mode), built-in DAQ session panel, and a "SHM Latency" tab that launches daoTimeDiff in its own tmux session and live-plots the AVG/RMS it publishes for any two SHMs picked from the file list), daoImDisp / daoImgDisp (lightweight single-image viewers), daoRTDMagic (OpenGL-textured, lowest-latency image RTD), daoRemoteShmViewer (mirrors a SHM stream from another host via daoRemoteShmFileServer.py)
DM control & display	daoDmCtrl (modal control of one DM channel through its M2A matrix — any modal basis, not analytic-only), daoDmChannelsCtrl (overview of all DM command channels at once, opens daoDmCtrl per channel), daoDmDisp / daoDmDispNoMap (read-only actuator display, mapped / raw bar chart)
Loop control	daoLoopDisp / daoLoopCtrl (generic open/close-loop and gain/leak control panels — point at any loop's state/gain/leak scalar SHMs, default lpCmd/lpGain/lpLeak), daoFastAndFuriousCtrl (runs daoFastAndFurious.py: basis, calibration, loop with live gain/leak/hold, test aberration; shows the PSF, the Strehl proxy and the correction phase)
Wavefront & slopes display	daoWfDisp / daoWfDispMap (reconstructed phase, plain or over an illumination map), daoSlopesDisp (X/Y slope quiver plot), daoShDisp (colour-coded slope grid), daoRTDMagicSH (low-latency Shack-Hartmann RTD), daoTtDisp (tip/tilt scatter monitor), daoBarDisp (generic real-time bar chart of any 1-D SHM vector)
Latency measurement	daoTimeDiffDisp (standalone SHM-to-SHM latency GUI — takes the two SHMs on the command line, Start/Stop launches daoTimeDiff in its own tmux session and live-plots its AVG/RMS window as a scatter + histogram; same measurement as daoShmViewer's "SHM Latency" tab, without the rest of the viewer)
Logging & telemetry	daoLogMonitor (live, filterable ZMQ log viewer), daoShmTelemetryConfigurator (graphical daoDAQ YAML editor, embedded in daoShmViewer)
Reusable widget	daoProcessWidget (live process-status widget — tmux session alive/dead, CPU load, last update — embedded in other GUIs)

Prerequiries

daoBase

daoBase should be installed. See https://github.com/Durham-Adaptive-Optics/daoBase

Required dependencies (CLI11, cfitsio, yaml-cpp, fmt, zmq, protobuf, ncurses)

waf configure fails without these - every app in apps/ links at least protobuf, zmq and CLI11.

Ubuntu/Debian:

sudo apt install libcli11-dev libcfitsio-dev libyaml-cpp-dev libfmt-dev libzmq3-dev \
                  libprotobuf-dev protobuf-compiler libncurses-dev

RHEL/Fedora/CentOS:

sudo yum install cfitsio-devel yaml-cpp-devel fmt-devel zeromq-devel \
                  protobuf-devel protobuf-compiler ncurses-devel

(CLI11 has no RHEL package as of writing; configure falls back to looking for a vendored CLI11.hpp/CLI/CLI.hpp header if pkg-config can't find it.)

macOS (Homebrew)

brew install openblas pkg-config protobuf cfitsio yaml-cpp fmt zeromq

Note: Homebrew installs OpenBLAS as openblas.pc. The waf configure script automatically checks for both blas and openblas pkg-config packages, so no additional configuration is needed.

Optional dependencies

waf configure auto-detects each of these and silently skips the app(s) that need it if missing - not having them is not an error, just a smaller build.

dependency	apt package	enables	skipped without it
BLAS (any implementation)	libopenblas-dev	daoMvM (CPU real-time matrix-vector multiply)	daoMvM
CUDA toolkit (nvcc + cudart)	see NVIDIA's install docs	daoMvMGPU, libdaoToolsGpu, daoGpuPipeline (also needs yaml-cpp), tests/testGpuStages	those
FFTW, single and double precision	libfftw3-dev	the FFT-based correlation centroider (daoToolsCorrFFT, daoComputeCentroidCorrelationFFT)	that centroider only - daoComputeCentroidCorrelation (the windowed-search version) is unaffected

Check waf configure's output for lines like BLAS detected: enabling BLAS build. / FFTW not found ... skipping FFT correlation centroider. to see what your machine actually got.

We recommand to use magicPlot Some of our plot tool uses magicPlot (optional)

pip install magicPlot

Real-time scheduling priority

Every real-time loop in apps/ requests SCHED_FIFO scheduling once at startup via daoThreadSetRtPriority(NULL, priority) (e.g. priority 93), so the OS doesn't preempt it for ordinary processes. Threads, CPU pinning, memory locking, mutexes and timed waits all go through daoBase's daoThread.h (daoThreadCreate, daoThreadPin, daoMemLock, daoMutex, daoSleepUntilNs): the programs no longer call pthread, sched_setscheduler, mlockall or clock_nanosleep themselves, so the same code builds where those don't exist (Windows). daoToolsSetRtPriority is kept, deprecated, as a call to it.

That request can be denied: Linux caps how high a priority a given user may request (the rtprio resource limit), and it defaults to 0 on most distros. daoThreadSetRtPriority tries the requested priority first, then the highest this user's rtprio limit allows, and logs which one it got (daoInfo on success, daoWarning on fallback or failure), so the operator always knows; it returns the priority set (0: normal).

Raising the limit (Linux)

Add a line to /etc/security/limits.conf (or a file under /etc/security/limits.d/), then start a new login session (log out/in, or a fresh login shell) for it to take effect:

<username>  -  rtprio  99

or for everyone in a group:

@rtgroup  -  rtprio  99

Confirm it applied with:

ulimit -r

Windows and macOS

daoThread.h has one implementation per platform (daoThreadPosix.c, daoThreadWin32.c in daoBase):

• Linux: everything above.
• macOS: no CPU pinning (a warning); SCHED_FIFO within the system's range. The macOS stand-ins for clock_nanosleep / sched_setscheduler in daoTools.c are kept, deprecated, for the programs of other repositories still calling them.
• Windows: a priority above 0 is THREAD_PRIORITY_TIME_CRITICAL in the HIGH priority class; pinning with SetThreadGroupAffinity; no whole-process memory lock (a warning).

Setting up a host for a real-time loop

docs/source/realtime.rst covers the rest, with what each step gave on one host: locked memory, GPU clocks, one core per real-time process, isolated CPUs, interrupts, idle states, real-time throttling. The scripts in scripts/:

• daoRtCheck [-c <cpus>] reports how the host is set up for the loop CPUs (changes nothing);
• daoToolSetCap gives the real-time binaries the capabilities for SCHED_FIFO and mlockall (an alternative to limits.conf);
• daoGpuClocks sets GPU persistence, locked clocks and the power limit (-n dry run, -u undo, -q query);
• daoRtPin (sourced by start scripts) places each process on its core: dao_rt_pin, dao_rt_cpu, dao_rt_core, dao_other_pin.

DM display

daoDmDisp.py lays out the channel displays followed by the combined command in three columns, adding rows as needed. Large grids scroll. The default is four channels; use -n / --channels for another count:

daoDmDisp.py -s dmCmd -m dmMap          # 4 channels
daoDmDisp.py -s dmCmd -m dmMap -n 6     # 6 channels
daoDmDisp.py -s dmCmd -m dmMap -n 8 --light
daoDmDisp.py -n 6 -l Flat,Loop,Turbulence,Pokes,Focus,TipTilt

For -n 6, the display reads dmCmd00 through dmCmd05, plus the combined dmCmd stream and the dmMap actuator map from /tmp/*.im.shm. These streams must already exist. Default labels are Flat, Loop, Turbulence, Pokes, then Channel 04, Channel 05, etc. Custom labels must supply one name per channel. The layout is generated in Python, so no separate .ui file is needed for each channel count.

daoBase API compatibility

The C/C++ tools use daoBase's primary shared-memory API (daoShmCreate, daoShmOpen, daoShmSetData, etc.). Build and install daoBase with that API (commit 8f64d74 or later) before rebuilding daoTools; older installed headers/libraries do not provide these names. Python callers keep the same public shm interface.

SHM names can be up to DAO_SHM_NAME_LEN - 1 characters (defined by daoBase's dao.h; with an older dao.h the tools fall back to the size of IMAGE.name). A longer name, or an SHM that cannot be opened, stops a tool with an error message. The GPU parts (daoMvMGPU with GPU SHMs, libdaoToolsGpu, daoGpuPipeline) need a daoBase with GPU SHMs (daoShmCreateGpu).

GPU pipeline

daoGpuPipeline -c <config.yaml> -L runs a chain of GPU stages — each the GPU version of a daoTools application, with the same result — in one process: the trigger SHM (usually the camera frame) starts one CUDA graph per frame, the intermediate data stays on the GPU (GPU SHMs), and every output SHM is still published each frame, so displays and telemetry keep working. -s N runs a single stage as its own process instead, to develop and tune stage by stage with the same configuration. Host SHMs read or written by the GPU should be in RAM (tmpfs, e.g. /dev/shm) so the GPU can use them in place. The outputs are published, and by default timestamped, at the end of the frame; -T (or stageTimestamps: true) timestamps each one when its data was ready on the GPU, to measure the latency of each stage, for about 10 µs more per frame. Stages, configuration and options: docs/source/gpu_pipeline.rst. Large parameters (a new reconstructor) can be uploaded in the background while the loop runs (paramUpload: async) instead of the frame waiting for the copy.

CPU pipeline

daoCpuPipeline -c <config.yaml> -L runs the same configuration on the CPU, in one process and one thread: each stage calls the libdaoTools function of its application, on the same host SHMs, with the same results bit for bit, but without a semaphore wake-up between stages. -W spins on the trigger, -C pins the loop thread, -w <cpus> splits the centroiders and the MVM over worker threads (still bit for bit), and plugins: loads stage types written outside daoTools (include/daoCpuStages.h). On a small Shack-Hartmann chain it is faster than three separate processes and than daoGpuPipeline; see docs/source/cpu_pipeline.rst.

See the migration regression tests for the exact rename mapping, before/after numerical checks, and validation limits.

Optional build dependencies

Configuration continues when optional C++ application dependencies are missing. The build prints the name of each skipped program and its missing or unusable dependencies:

Program	Required optional dependencies
daoDAQ	CFITSIO, CLI11, fmt, ZeroMQ, protobuf, yaml-cpp
daoDownsample	CLI11, protobuf, ZeroMQ

CFITSIO must provide the unsigned 64-bit constants used by daoDAQ; configuration checks both its headers and library. CLI11 must provide CLI/CLI.hpp. The core C library and other tools remain enabled. Existing BLAS, CUDA, and FFTW checks continue to control their respective targets.

After installing a missing dependency, rerun waf configure and waf build to enable the corresponding programs. Skipping a program does not uninstall an older copy that is already present.

Build

waf configure --prefix=$DAOROOT
waf
waf install

Build Documentation

Run waf build_docs to build the Sphinx HTML documentation (outputs to docs/build/html).

waf build_docs

To clean the built documentation:

waf clean_docs

Example

It is possible to create just the shared memory for an example. See examples below

staring a simple clock

The first example is a simple C code increasing a counter in the shared memory. This software clock can be used to synchronize different program using built-in semaphore in the DAO SHM

create SHM

can be skipped if SHM already created

import daoShm
import numpy as np
clockShm=daoShm.shm('/tmp/demoClockShm.im.shm',np.zeros((1,1)).astype(np.uint32))

run example

Now let's run the clock at 1.5kHz in a new console

daoClock -L demoClockShm 1500

writing data in shared memory

This example is a C program writing in the shared memory at a specific rate. The program is waiting for a new value in another shared memory. We can use the previous example and use the clock as the trigger

create SHM

can be skipped if SHM already created

import daoShm
import numpy as np
shm=daoShm.shm('/tmp/demoShm.im.shm',np.zeros((100,100)).astype(np.float32))
clockShm=daoShm.shm('/tmp/demoClockShm.im.shm',np.zeros((1,1)).astype(np.uint32))

run example

Now run in a new console the program writing at the clock rate in the share memory random values

daoRandWriterSync -L demoShm demoClockShm