PAPER DEEP DIVE
AEGIS: Assay-Aware Protocol Validation and Runtime Monitoring for Open-Source Liquid Handling Robots
Self-driving laboratories increasingly rely on low-cost liquid handlers such as the Opentrons OT-2, which ship without the pressure-based aspiration monitoring of Hamilton or Tecan systems and are typically run open-loop. Two failure modes go undetected: protocols that are syntactically valid but violate assay-specific invariants (e.g., tip reuse between a PCR template and a no-template control), and physical execution failures (partial dispense, air bubbles, missing tips) at runtime. We present AEGIS, a two-layer guardian for both. Layer 1 pairs a curated machine-readable assay rule database with an LLM that reasons over OT-2 Python code, reaching an adjusted F1 of 0.97 on a 24-protocol benchmark across five assay families and beating rules-only and LLM-only ablations across five backends; a free open-weight model ties the best proprietary one, so no paid API is required. Layer 2 fits a PCA world model to YOLO-cropped four-frame pipette trajectories; under a leakage-free leave-one-plate-out evaluation it reaches average precision 0.89 and operating-point F1 0.71 (AUROC 0.80), a deployment-faithful number that matches the live demonstration, and we characterize the small-pipette (p20) resolution limit (F1 0.47). A live demonstration on a physical OT-2 (five replicates per condition) catches planted no-tip failures deterministically and partial dispense on coloured dyes, with an always-VLM self-vote gate lifting partial-dispense recall to 5/5; transparent water is a principled limit of any front-view-only monitor, which AEGIS surfaces as low-confidence VLM reasoning rather than a wrong verdict. Cascade triage holds VLM cost near $1.63 per plate versus $10.33 for an always-VLM baseline. AEGIS is open source and, to our knowledge, the first system to unify pre-flight assay-aware validation with runtime visual monitoring for an open-source liquid handler.
Paper: AEGIS: Assay-Aware Protocol Validation and Runtime Monitoring for Open-Source Liquid Handling Robots
Authors: Priyanka V. Setty, Arvind Ramanathan, Ian Foster, Rick Stevens (Argonne National Laboratory / University of Chicago)
Source: arXiv:2607.15620
1. Background and Motivation
Self-driving laboratories (SDLs) are reshaping experimental science by closing the design–build–test–learn loop: predictive models propose experiments, robotic platforms execute them, and the resulting data updates the model in a continuous loop. However, the scientific integrity of this loop depends on whether each wet-lab step actually does what the protocol intended — at scale, no human can audit every well. A silently failed wet-lab run does not produce an error; it produces corrupted data that compounds across subsequent model updates.
Figure 1: AEGIS guards against two silent failure modes — assay-level (semantic) errors in the protocol code, and physical execution failures inside the liquid handler during the run. Both propagate as bad data into downstream analyses and predictive models.
The key problem: the robots themselves are "biologically blind" — a liquid handler executes whatever Python protocol it is given, with no awareness of assay-level intent. Two distinct failure modes silently corrupt runs:
- Assay-level (semantic) errors: protocol violations that pass every syntactic check, e.g., tip reuse between PCR template and no-template control (NTC), wrong dilution direction, drug only in treated wells.
- Physical execution errors: partial dispense, air bubbles, missing tips — runtime failures the robot's own log will not flag.
2. AEGIS Two-Layer Architecture
Figure 2: AEGIS two-layer architecture. Layer 1 reads the OT-2 protocol's Python source before the run and flags assay-level violations against a curated rule database. Layer 2 watches the pipette through a camera, scores per-well trajectories with a PCA world model trained on normal trajectory images, and routes ambiguous wells to a VLM that arbitrates pause decisions.
AEGIS comprises two guardian layers:
- Layer 1 (Pre-flight Validator): reads the protocol's Python source, checks against 22 curated assay rules, flags violations.
- Layer 2 (Runtime Visual Monitor): watches the pipette via camera, scores per-well trajectories with a PCA world model, routes uncertain wells to a VLM gate for pause arbitration.
3. Layer 1: Assay Rules and Validator
Layer 1 formalizes 22 assay-specific rules in a machine-readable JSON schema, each with an identifier (e.g., PCR-2: "fresh tip between every PCR template and the NTC"), natural-language statement, severity level, assay scope, and optional structural indicators. Rules cover contamination (tip-reuse patterns), ordering (low-to-high standards, dilution direction), completeness (master-mix presence, mixing after each dilution step), and source/destination correctness.
The validator itself is a single-pass LLM call consuming the rule database, a curated Opentrons API reference, and the protocol's Python source. The LLM is asked for (i) the inferred assay family with confidence label, (ii) a rule-by-rule evaluation listing offending wells and citing each violated rule, and (iii) a final pass/fail verdict.
The design is deliberately hybrid: rules tell the LLM what to check; the LLM figures out how to check it in the code. Ablations compare three configurations: full hybrid (rules+LLM), LLM-only (rules withheld), and rules-only baseline. Five backends span capability tiers: OpenAI o4-mini, Claude Opus 4, GPT-4o, Claude Sonnet 4.5, and NVIDIA Nemotron-3 Ultra 550B (open-weight).
Layer 1 Performance
On the 24-protocol benchmark (five assay families), the adjusted $F_1$ reaches 0.97, outperforming both rules-only and LLM-only ablations across all five backends. The open-weight model (Nemotron) ties the best proprietary model, proving the approach does not depend on a paid API.
4. Layer 2: Runtime Visual Monitoring
4.1 Data Collection
A Logitech webcam captures the OT-2 deck at 1 fps with a black cardboard background. The system reads the Opentrons HTTP API for each pipetting command's completion timestamp and aligns captured frames. The offline evaluation collected 64 p1000 well trajectories: 26 physically normal and 38 planted failures, across three reagent contrasts (red, yellow, water) and four failure types (partial dispense, no dispense, no tip, air bubble). Each trajectory consists of four keyframes: aspirate, transit, dispense, post-dispense.
4.2 YOLOv8 Tip Detection
Figure 3: YOLOv8-nano tip detection. Left: raw camera input; Right: fine-tuned detector returns a tight bounding box around the pipette tip. The detected region is cropped, padded, and resized for the PCA world model.
YOLOv8-nano was fine-tuned on 69 hand-labelled frames, reaching mAP50 = 0.995. The detector performs geometric tip localization only and is agnostic to whether a well is normal or failing.
4.3 PCA Trajectory World Model
For each well, AEGIS concatenates YOLO-cropped tip patches from the four keyframes (greyscale, $48\times 48$ each) into a 9,216-dimensional trajectory vector. PCA with $k=15$ components is fit on normal trajectories, retaining 97.7% of variance. Reconstruction error $\|x-\hat{x}\|_{2}^{2}$ serves as the anomaly score. The choice of linear PCA is deliberate: at $n\sim 26$ normal trajectories in 9,216 dimensions, deeper alternatives (autoencoders, DreamerV3, IRIS) learn near-identity reconstructions that reproduce failures as faithfully as normals.
$$\text{anomaly score} = \|x - \hat{x}\|_{2}^{2}, \quad \hat{x} = \text{PCA reconstruct}(x)$$4.4 Cascade and always-VLM Operating Points
Figure 4: Layer 2 live cascade. Per-well events flow from the camera and OT-2 HTTP API into PCA triage, then route to VLM arbitration for pause decisions.
The reconstruction-error scale drifts by roughly a factor of two between sessions. AEGIS does not re-fit PCA or retune a global threshold — instead it calibrates the cascade per run on the first six wells (standards, controls, normalisation reagents that are physically normal by design). Recording PCA scores $c_{1},\ldots,c_{6}$, the silent-pass band is defined as:
$$\text{band} = \mathrm{median}(c_{i}) \pm 0.7 \cdot \mathrm{MAD}(c_{i})$$For each post-calibration well, the cascade emits one of three routing decisions:
- Silent-pass: PCA score inside the per-run calibration band — no API call.
- Blank floor: score below $10^{-4}$ — YOLO produced no tip-containing crop, the dominant signature of missing-tip failures, routed to VLM.
- Gray zone: any other score outside the band — trajectory routed to VLM with cropped frames.
The PCA scorer is never permitted to trigger a robot pause on its own. The VLM acts as a second-stage validator, the sole arbiter of any pause decision: PCA triages, the VLM decides. This eliminates a class of false positives where an unusual but valid well would halt a working run.
Routed wells enter a Claude Opus 4.7 vision call with the four cropped trajectory frames and a reagent-colour-aware system prompt structured as three in-call assessments (skeptical, routine, expert lenses). The aggregator applies an any-fail rule: if any lens returns fail, the overall verdict is fail.
5. Method Architecture Flow
flowchart TD
subgraph L1["Layer 1: Pre-flight Validation"]
P["OT-2 Python Protocol"] --> R["22 Assay Rules DB"]
R --> V["LLM Validator\n(rules + API ref + code)"]
V --> V1{"Assay-level\npass/fail?"}
V1 -- fail --> H1["Halt before run"]
V1 -- pass --> R2["Proceed to run"]
end
subgraph L2["Layer 2: Runtime Visual Monitor"]
C["Camera @1fps"] --> Y["YOLOv8-nano\ntip detection"]
O["OT-2 HTTP API\ncommand timestamps"] --> F["4-keyframe trajectory\n(aspirate/transit/dispense/post)"]
Y --> F
F --> PCA["PCA World Model\nk=15, 9216-dim"]
PCA --> S{"Score routing"}
S -- silent-pass --> OK["No API call"]
S -- blank-floor --> VLM2["VLM gate\n(3-lens self-vote)"]
S -- gray-zone --> VLM2
VLM2 --> D{"Any-fail verdict?"}
D -- fail --> PAUSE["Pause OT-2 via HTTP API\n(effective halt ~13.3s)"]
D -- pass --> OK
end
R2 --> L2
6. Experimental Results
6.1 Layer 1: Protocol Validation
| Backend | Type | Adjusted F1 |
|---|---|---|
| OpenAI o4-mini | Reasoning model | 0.97 |
| Claude Opus 4 | Frontier model | 0.97 |
| GPT-4o | Frontier model | 0.97 |
| Claude Sonnet 4.5 | Mid-tier model | 0.97 |
| NVIDIA Nemotron-3 Ultra 550B | Open-weight | 0.97 (ties best) |
6.2 Layer 2: Offline Evaluation (LOPO)
Under the leakage-free leave-one-plate-out (LOPO) protocol, Layer 2 achieves average precision (AP) 0.887 [0.799, 0.949], AUROC 0.804, operating-point $F_1$ = 0.708 (precision 0.793, recall 0.639) at failure prevalence 0.58. The LOPO result closely matches the deployment chromophoric-cascade recall of ~0.6, so the offline number predicts deployment performance.
Figure 5: Per-trajectory PCA reconstruction error on the p1000 set, grouped by ground-truth class. With no class labels at training time, normal trajectories concentrate at low reconstruction error while failures spread toward higher error.
The per-failure-mode breakdown shows a clean dose-response — the more liquid a partial dispense leaves in the tip, the more reliably it is caught:
| Failure type | Recall (LOPO) |
|---|---|
| Partial dispense, 25% dispensed (75% retained) | 3/3 (100%) |
| Partial dispense, 50% dispensed | 2/3 (67%) |
| Partial dispense, 75% dispensed (25% retained) | 1/3 (33%) |
| No dispense | 5/6 (83%) |
| Air bubble | 1/6 (17%) |
| Total | 23/36 (64%) |
Detection is essentially colour-independent (red and water both 8/12, yellow 7/12) because the scorer keys on tip meniscus and trajectory geometry. The p20 channel marks the resolution limit: AP 0.771, AUROC 0.666, $F_1$ 0.468 — the p20 tip occupies too few pixels.
6.3 Live Demonstration
The cascade was exercised end-to-end on a physical OT-2, five consecutive runs per condition:
| Failure × Reagent | Default cascade | always-VLM |
|---|---|---|
| Partial dispense × red dye | 3/5 [0.23, 0.88] | 5/5 [0.57, 1.00] |
| Partial dispense × yellow dye | 3/5 [0.23, 0.88] | 5/5 [0.57, 1.00] |
| Partial dispense × water | 0/5 [0.00, 0.43] | 1/5 [0.04, 0.62] |
| No-aspirate × red | 3/5 | 2/5 |
| No-tip × red | 5/5 | 5/5 |
Effective-halt latency (failed dispense to OT-2 actually stopping) is $13.27\pm 0.47$ s ($n=9$ paused runs): monitor-side ~9.5 s (4 s frame-flush + 5.5 s VLM call), plus the OT-2 applying pause at the next protocol boundary. The cascade triage holds per-plate VLM cost at $1.63 versus $10.33 for always-VLM.
Figure 6: Latency decomposition from failed dispense to OT-2 effective halt. Monitor-side ~9.5 s; effective-halt latency 13.27±0.47 s (n=9).
7. Key Insights and
观测向量定义
$$ \mathbf{o}_{t}=\left[\boldsymbol{\omega}_{t},\;\mathbf{g}_{t},\;\mathbf{c}_{t},\;\boldsymbol{\theta}_{t},\;\dot{\boldsymbol{\theta}}_{t},\;\mathbf{a}_{t-1}\right]^{\top} $$
目标关节位置
$$ \boldsymbol{\theta}_{t}^{\mathrm{target}}=\boldsymbol{\theta}^{\mathrm{default}}+\mathbf{a}_{t} $$
PD控制力矩
$$ \boldsymbol{\tau}_{t}=K_{p}(\boldsymbol{\theta}_{t}^{\mathrm{target}}-\boldsymbol{\theta}_{t})-K_{d}\dot{\boldsymbol{\theta}}_{t} $$
Limitations- Detection is failure-mode-dependent, which is why the cascade has two stages: partial dispense is a VLM strength (retained column is a direct cue), no-aspirate is a PCA mode (empty tip after dispense is ambiguous to VLM but the transit frame is not), no-tip is deterministic via BLANK_FLOOR. Routing everything to VLM would lose no-aspirate; trusting PCA alone would forfeit precision.
- Transparent reagents fail for two reasons: geometric (transparent fluid in a transparent tip seen from the side gives almost no contrast) and resolution (consumer webcam puts too few pixels on the tip). The robust fix is a post-run top-down plate view (prospective Layer 3).
- The cascade improvement is multi-factor: the redesign changed calibration count, silent-pass band, and fail-well position simultaneously, so the gain reads as their joint effect.
8. Conclusion
AEGIS is the first system to unify pre-flight assay-aware validation with runtime visual monitoring for an open-source liquid handler. Its two-layer architecture achieves an adjusted $F_1$ of 0.97 at the protocol level via a hybrid rules+LLM validator, and 0.887 AP with 0.71 operating-point $F_1$ at runtime via a PCA-triage + VLM-arbitration cascade, with effective-halt latency of $13.27\pm 0.47$ s and per-plate VLM cost of only $1.63. The core design pattern — cheap triage with an expensive validator, neither layer acting alone — generalises beyond AEGIS to any deployment with a strong but imperfect cheap model and a more capable but fallible LLM gate. The authors release the 24-protocol benchmark, 22 rules, YOLO weights, trajectory dataset, and live demo harness under the MIT license, lowering the floor for adding a guardian layer to any community OT-2 setup — there is little reason for a new open-source self-driving laboratory to operate fully open-loop.
SOURCE LINKS



