OpenVLHarness: Real World Visual Agentic Tool-Use with Multimodal Memory

Paper metadata

Title: OpenVLHarness: Real World Visual Agentic Tool-Use with Multimodal Memory

Authors: Yu Zhou, Cheng-Fu Yang, Rui Sun, Di Wu, Sibo Peng, Mingyang Zhang, Zhen Yang, Ruiyi Zhang, Nanyun Peng, Zhe Gan, Kai-Wei Chang

Links: arXiv:2610.26860v1 (submitted 19 Oct 2026); code and evaluation kit released at github.com/uclanlp/OpenVLHarness, with model and dataset mirrors under Hugging Face uclanlp/OpenVLHarness.

Code status: Released. The repository mirrors Section 2 of the paper: the capability layer lives in openvlharness/capabilities/, the memory registry in openvlharness/memory/, and the benchmark harness in evaluation_kit/ plus run_benchmark.py.

One-sentence summary

OpenVLHarness reframes multimodal reasoning from "can the model think harder" to "can the model obtain evidence": a capability layer standardizes nine tool interfaces and renders their outputs into model-readable observations, while persistent multimodal memory keeps every intermediate artifact addressable, lifting 22 real-world visual benchmarks by 9.7 to 12.6 points over matched no-tool baselines.

Background and motivation

Multimodal large language models have become competent readers of images, yet many real questions cannot be answered by reading alone. How far apart are the centers of the window and the cabinet? That demands camera poses, metric depth per view, and object localization, followed by a coordinate transformation. Which awards did this backpack win? That demands zooming into a tiny logo, reading it, and verifying the claim on the web. Such evidence lives outside the parameters and outside a single forward pass; it must be acquired step by step with external tools.

The authors first rule out the "just think longer" intuition with a controlled comparison. On the full 22-benchmark suite, GPT-6 Luna alone moves from 44.3 at low reasoning effort to 48.0 at maximum effort, and Visual Sketchpad from 44.7 to 48.8, while the model-token bill per thousand questions climbs from USD 0.58 to USD 3.17. Internal reasoning hits diminishing returns quickly; the evidence gap does not close by itself no matter how long the model deliberates.

Existing tool-use frameworks do not close it either. Systems such as OctoTools and PixelReasoner organize manually designed tool interfaces and execution strategies, and they typically feed raw tool output back into the context: hundreds of unannotated boxes, an unlabeled mask image, an intermediate result with no handle. The model must both decode heterogeneous formats and remember what it saw two turns ago, and both are frequent failure modes.

The paper therefore poses three design questions. How can heterogeneous backends (segmentation, OCR, depth, search engines) share one calling convention? How can tool output be rendered into observations the model can read and cite? How can images, structured data, and web references produced during a single query persist across turns instead of evaporating?

OpenVLHarness answers with two layers and one memory bus: a capability layer that handles invocation and rendering, a persistent multimodal memory that handles retention and addressing, and a coding interface that computes over retained data. Together they form an iterative evidence workflow: acquire, inspect, compose, until the evidence supports an answer.

Preliminaries

Reasoning harness. A harness is the scaffolding around the model: tool interfaces, executors, context management, evaluation protocol. The paper's stance is that the harness is a first-class citizen. The same backbone can score more than ten points differently under a different harness, so the harness deserves systematic design and optimization rather than ad hoc assembly.

Capability. A capability is defined by the operation it provides, its typed inputs, and the evidence and reusable data it exposes. Visual grounding, for instance, accepts an image reference and a textual query and exposes object locations, metadata, and a visualization; a supporting backend may additionally provide masks. Because the definition lives at the interface level, backends are swappable: grounding can run on SAM 3 or Grounding DINO, OCR on PaddleOCR-VL or DeepSeek-OCR, depth on Depth Anything 3 or VGGT, while the orchestrator's calling convention stays fixed.

Evidence rendering. Instead of piping tool stdout back verbatim, evidence rendering rewrites backend output into model-friendly observations: grounding returns bounding boxes on a 0-1000 scale together with a labeled mask image whose numbered overlays correspond one-to-one to the object indices in text. The model can point at the picture instead of pattern-matching across hundreds of floats.

Method in detail

Architecture: orchestrator, capability layer, memory

The running unit of OpenVLHarness is an MLLM orchestrator surrounded by capabilities. The orchestrator only decides which capability to call, with which arguments, and when to commit to an answer. The capability layer translates each call into a backend request and renders the backend output into model-readable evidence. The memory layer retains every artifact for the duration of the query. Figure 1 shows the loop: the user question and input images enter the orchestrator, tool calls descend through the capability layer to models and services, rendered evidence and an Environment Update flow back, and the registry on the right stays addressable at all times.

Figure 1: Overview of OpenVLHarness. The orchestrator invokes capabilities through typed interfaces; the capability layer connects them to backend models and services and renders outputs into model-facing evidence (labeled mask visualizations plus structured bindings); persistent multimodal memory retains image artifacts, structured data, and web references.

The base instantiation ships nine capabilities: five for perception (Zoom-in, Visual Grounding, OCR, Depth Estimation, Camera Trajectory), three for information seeking (Text Search, Image Search, Webpage Visit), and a Python Coding Agent for computation over intermediate results. All nine share one calling convention, so adding a capability means implementing one interface.

Capability layer: typed interfaces and evidence rendering

The responsibility of the capability layer is stated plainly in openvlharness/capabilities/_base.py: BaseCapability validates task calls, resolves artifact references, executes tools, and renders model evidence. Images in call arguments must be given as memory handles such as input_image_3; the layer resolves the handle to a real file before invoking the backend, which removes fabricated paths at the interface level.

Rendering decides what the model sees. For grounding, the returned text states how many objects were grounded and their boxes on the 0-1000 scale, accompanied by a numbered mask image; selected structured results are written into memory through store_selected_binding so later turns can fetch them by name. OCR returns confidence-tagged text blocks, depth estimation returns an indexable depth-map handle, camera trajectory returns intrinsics and extrinsics. Everything enters the context as a concise observation plus an addressable artifact.

Persistent multimodal memory: three artifact classes

The registry holds three classes of artifacts. Image artifacts: user inputs as input_image_n and tool outputs as tool_generated_image_n, including crops, mask visualizations, and depth maps. Structured data bindings: boxes and masks, depth maps, and camera trajectories stored as named dictionaries such as segment_cabinet or depth_result. Web references: pages visited during search as webpage_url_n, available for re-inspection. The implementation is split across memory/store.py, memory/images.py, memory/bindings.py, and memory/urls.py.

Memory is not a passive warehouse. After every tool turn, EnvironmentUpdateBuilder in memory/environment_update.py emits an environment update message prefixed with "Environment Update (not a new question):" that lists currently available image IDs and data bindings and explicitly instructs the model: "Do NOT invent image file names or paths." The update is injected as a system-style message rather than a new user question, so the model always knows which artifacts it holds and never misreads the update as a fresh task.

Coding interface: computation over memory

Many answers are not a verbatim repeat of one tool output but a composition of several. The Coding Agent executes generated Python over memory bindings: it can read intrinsics and extrinsics from camera_trajectory_result, metric depth from depth_result, boxes from segment_window, back-project pixels into world coordinates, and measure distances. The first panel of Figure 2 is exactly this chain: camera trajectory supplies a shared coordinate frame, depth supplies metric scale, grounding supplies the window and cabinet masks, and a short program returns 1.1957 meters.

Figure 2: Three real task chains. (a) Camera trajectory, depth, and grounding combined in code to measure a 3D distance. (b) Grounding, zoom-in, OCR, image search, text search, and webpage visit to verify an award claim. (c) Two grounding calls plus code for pill counting.

Written as formulas, the computation in Figure 2(a) back-projects a pixel through intrinsics $K$ and extrinsics $(R, t)$ as $P_{\mathrm{world}} = R^{\top}(P_{\mathrm{cam}} - t)$, with $P_{\mathrm{cam}}$ assembled from pixel coordinates and metric depth, and the answer is the norm of the difference between object centers, $d = \|P_{\mathrm{world}}^{(w)} - P_{\mathrm{world}}^{(c)}\|_2$. The counting example is set algebra: 134 masks for "pill", plus the 5 of 72 "round pill" masks not already covered, $N = |\mathcal{B}_{\mathrm{pill}}| + |\mathcal{B}_{\mathrm{round}} \setminus \mathcal{B}_{\mathrm{pill}}| = 134 + 5 = 139$. Tools gather evidence; code does the arithmetic; the model wires the two together.

The iterative evidence loop

Assembled, a single query is a loop: the orchestrator proposes a tool call, the capability layer executes and renders it, memory registers the artifact and broadcasts an environment update, and the orchestrator decides the next move on the updated context: gather more evidence, run code, or commit to an answer. The diagram below follows Section 2 and the released implementation.

flowchart TD
  Q["User question and input images
input_image_1 .. input_image_n"] --> O["MLLM orchestrator"]
  O -->|"typed tool call"| C["Capability layer, nine capabilities
Zoom-in Grounding OCR Depth CameraPose
TextSearch ImageSearch WebVisit CodingAgent"]
  C --> B["Backend models and services
SAM 3 Grounding DINO PaddleOCR-VL
Depth Anything 3 VGGT search engines"]
  B --> E["Evidence rendering
numbered masks, boxes, confidence, bindings"]
  E --> M["Persistent multimodal memory
image artifacts, data bindings, web refs"]
  M --> U["Environment Update
broadcast available image IDs and bindings"]
  U --> O
  M --> K["Coding Agent
programs over stored bindings"]
  K --> O
  O --> A["Final answer"]

Self-optimizing the harness: search and tool generation

The paper does not stop at a hand-built harness; it adds two automatic optimization routes. The first is harness search (Algorithm 1): maintain a population $\mathcal{P}$ initialized with the manual baseline $h_0$, scored as $\mathrm{score}(h_0) \leftarrow \mathrm{Eval}(h_0, \mathcal{D}, S)$. Each round analyzes failures $a \leftarrow \textsc{Analyze}(p, \mathcal{F}, \mathcal{D})$, proposes edits $\{\delta_1, \dots, \delta_M\} \leftarrow \textsc{Propose}(p, a)$, composes candidates $c \leftarrow p \oplus \{\delta_i\}_{i \in I}$ with $|I| \leq K$, and keeps the best $\mathcal{P} \leftarrow \mathrm{Top}_N(\mathcal{P} \cup \mathcal{C}, \mathrm{score})$. The reported configuration is $N=4$, $M=4$, $K=4$, sample size $S=3$, three rounds.

The second route is tool generation (Algorithm 2): cluster failures into domains and subsets $\{(d_j, \mathcal{S}_j)\}_{j \leq D} \leftarrow \textsc{Analyze}(\mathcal{F})$ with $D \leq 5$, run a candidate tool $\tau$ on a tuning subset $\rho \leftarrow \mathrm{Run}(h \cup \{\tau\}, \mathcal{S}_j^{\mathrm{tune}})$, then measure lift $(\mathrm{fix}, \mathrm{broke}) \leftarrow \mathrm{Lift}(h \cup \{\tau\}, \mathcal{S}_j)$. The tool is accepted only if $\mathrm{fix} \geq 2$ and $\mathrm{fix} > \mathrm{broke}$ on the subset and $\mathrm{fix}' \geq 1$ and $\mathrm{fix}' > \mathrm{broke}'$ on a held-out split, with at most $T=5$ candidates per domain. The double gate rejects pseudo-improvements that fix two examples while breaking three others.

flowchart LR
  P["Population P with manual baseline h0"] --> EV["Eval scoring"]
  EV --> AN["Analyze failed trajectories"]
  AN --> PR["Propose four candidate edits"]
  PR --> CO["Compose candidates, at most K edits"]
  CO --> TP["Top-N selection keeps population"]
  TP --> AN
  AN --> CL["Cluster failures into domains and subsets"]
  CL --> RU["Run candidate tool on tuning subset"]
  RU --> LF["Lift test counts fix and broke"]
  LF --> GT["Gate: fix at least 2 and fix above broke
held-out split must pass too"]
  GT --> NH["New tool joins the harness"]

The division of labor between the two routes follows the failure profile. On ERQA and RealWorldQA, failures are mostly evidence-interpretation errors (71% and 56% respectively), and harness search adds +4.1 and +3.7 points while tool generation adds only +0.2 and +0.1. On SparBench and FREAK-VQA, failures are mostly missing-tool errors (channels A through C account for 75% and 62%), and tool generation adds +8.2 and +4.5 points while search slightly hurts. The optimizer reads what failures look like before deciding whether to edit instructions or build tools.

Experiments

Main results: 22 benchmarks, eight backbones

Table 1 summarizes averages over the 22 benchmarks. Two signals stand out. First, the gains are systemic: Qwen3-VL-8B rises from 34.24 to 45.41 (+11.2), Qwen3-VL-32B from 38.30 to 50.92 (+12.6), GPT-5, GPT-6 Sol, and Kimi K3 gain +11.3, +11.6, and +9.7, and GPT-6 Astra at medium effort moves from 60.27 to 70.17. Second, tool interfaces alone are not enough: OctoTools and Visual Sketchpad score 31.80 and 30.49 with the 8B backbone, below the no-tool baseline, showing that tool access without evidence rendering and memory actively hurts.

Backbone	No-tool baseline	OctoTools	Visual Sketchpad	OpenVLHarness
Qwen3-VL-8B	34.24	31.80	30.49	45.41
Qwen3-VL-32B	38.30	-	-	50.92
Qwen2.5-VL-7B	25.29 (AdaReasoner) / 28.02 (PixelReasoner) / 25.02 (DeepEyesV2)	-	-	33.45
GPT-5	44.4	-	-	55.7
GPT-6 Sol	52.7	-	-	64.3
Kimi K3	56.4	-	-	66.1
GPT-6 Luna (medium effort)	45.8	43.8	45.5	56.5
GPT-6 Astra (medium effort)	60.27	-	-	70.17

Table 1: Averages over 22 benchmarks (summary of paper Tables 1 and 2). The Qwen2.5-VL-7B baseline cell lists training-based methods AdaReasoner, PixelReasoner, and DeepEyesV2.

What reasoning budget cannot buy

Table 2 puts "more thinking" and "more harness" on the same invoice. At medium effort, OpenVLHarness averages 56.5, above the maximum-effort scores of every competitor: 48.0 (Base), 48.6 (Codex), 48.8 (Visual Sketchpad), a margin of 7.7 to 8.5 points. Its cost is USD 0.92 per thousand questions, about 29% of Base at max effort (USD 3.17) and about 18% of Visual Sketchpad at max effort (USD 5.15), i.e. at most 37% of the competitors' maximum-effort token cost. At low effort the asymmetry is starker: OpenVLHarness scores 49.92 for USD 0.64, already above every competitor's max-effort average.

Method (GPT-6 Luna)	Low effort	Medium effort	Max effort	Cost at medium (USD/1k)	Cost at max (USD/1k)
Base (bare model)	44.3	45.8	48.0	0.58	3.17
+ Codex	36.9	43.8	48.6	1.29	2.52
+ Visual Sketchpad	44.7	45.5	48.8	1.89	5.15
+ OpenVLHarness	49.92	56.5	60.1	0.92	2.99

Table 2: Reasoning effort versus harness (summary of paper Table 2, full evaluation splits of the 22 benchmarks).

Ablation: what each layer is worth

Table 3 accumulates components as described in Section 3.4. Raw tool access adds +3.8 and +4.3 points at 8B and 32B, concentrated in grounded perception. The capability layer, which restores full tool schemas and renders heterogeneous outputs into concise observations, adds another +3.4 and +3.2. Textual memory (data bindings, web references, environment updates, but tool-generated images only as descriptions) adds +0.7 and +0.3. Completing persistent multimodal memory with visual attachments and reusable image handles adds a further +3.1 and +4.5, reaching 47.1 and 52.5 overall; this last step matters most in visual search and deep research, exactly the domains that need to look back at previously generated images.

Cumulative variant (delta over previous row)	Qwen3-VL-8B	Qwen3-VL-32B	Domains benefiting most
No-tool baseline (overall average)	37.06	41.29	-
+ raw tool access	+3.8	+4.3	grounded perception
+ capability layer (interfaces + evidence rendering)	+3.4	+3.2	grounded perception, spatial
+ textual memory (bindings, web refs, updates)	+0.7	+0.3	visual search and deep research
+ full multimodal memory (visual attachments, handles)	+3.1	+4.5	visual search and deep research
Full harness (overall average)	47.14	52.51	-

Table 3: Ablation study (paper Table 3 and Section 3.4 prose, averaged over three runs).

Failure analysis: what 9,893 critiqued trajectories say

The authors used Kimi K3 to critique 9,893 failed Qwen3-VL-8B trajectories; the critic's labels agree with human annotations on 92% of 100 sampled failures. The failure profile splits by domain. In grounded perception, 75% of failures are incorrect tool output, mostly in crowded scenes with more than 25 objects, beyond exhaustive detection. In visual search and deep research, 49% of failures are omitted tool calls. Failures where a correct tool output was unusable or ignored account for only 3-8%, indicating that the capability layer generally does deliver evidence to the model; the bottleneck is acquisition and call strategy, not presentation.

Figures 3 to 5 show the raw artifacts of the counting example: the input image, the 134 numbered masks for "pill", and the 72 numbered masks for "round pill". The code stage finds that 67 of the 72 round pills are already covered by the "pill" set and 5 are new, yielding 139. The example demonstrates both why memory matters (both grounding results must survive) and why rendering matters (numbered masks let code verify coverage by index).

Figure 3: Input image of the FSC-147 counting example (paper Appendix F).

Figure 4: Visual Grounding returns 134 numbered masks for "pill"; the numbers are object handles the model can cite.

Figure 5: The same image with 72 numbered masks for "round pill", aligned with the numbering of Figure 4 so code can perform set algebra.

Limitations

Author-stated: crowded scenes remain a hard boundary. Exhaustive detection breaks beyond roughly 25 objects, and the dominant failure mode in grounded perception traces back to exactly this; the paper concedes that the current capability layer offers no hierarchical or region-based detection strategy.

Author-stated: gains depend on backbone proficiency. In general VQA and spatial reasoning, the harness can deliver evidence but cannot make a backbone understand it; the critique analysis shows these domains fail more on reasoning errors than on acquisition errors.

Author-stated: critiques cover failed trajectories only. The 9,893 critiques provide diagnoses and remedy directions, not predictions of recoverable accuracy; proprietary backbones were run once due to API cost, so no variance estimates exist for them.

Our assessment: latency and operational cost are unaddressed. A single query entails multiple model calls plus inference across several backend services, yet the paper reports token bills without end-to-end latency. Deploying and versioning nine capability backends (segmentation, OCR, depth, pose, search, coding) in production is a real cost that the evaluation protocol does not capture.

Conclusion and outlook

The contribution of OpenVLHarness compresses into one line: turn tool outputs into model-readable observations, turn intermediates into addressable memory, and turn the harness itself into a searchable object. Average gains of 9.7 to 12.6 points across 22 benchmarks and eight backbones, plus a cost curve where a medium-effort harness beats competitors at maximum effort for about a third of their tokens, suggest that the next tier of multimodal reasoning gains will come from engineering structure around the model rather than from larger reasoning budgets.

For robotics researchers the interface shape matters as much as the scores: typed capabilities, named artifacts, and broadcast environment updates are precisely the primitives an embodied agent needs to manage intermediates from multiple sensors. Swap depth maps and poses for point clouds and joint states, swap web search for manual lookup, and the same memory bus ports into a robot stack almost unchanged. Hierarchical perception for crowded scenes, failure-driven tool generation, and joint scheduling of harness and reasoning budget are the visible next steps on this route.

Golden quote

"Evidence does not appear because the model thinks longer: rendering tool output into model-readable observations and retaining intermediates as addressable memory is the real distance between a multimodal agent and a bare model."