PAPER DEEP DIVE
PaperBanana-Interact: Scientific Diagram Refinement with Multi-Turn Human Feedback
Recent efforts have aimed to automate scientific diagram generation from paper content (Lin et al., 2026; Zhu et al., 2026a). However, fully satisfying an author's visual and communicative preferences in a single turn is challenging: in our formative user study (N = 14), all participants requested further revisions after viewing an initial draft, and 86% of them rated the refined diagrams as more satisfactory. Despite the clear demand, the multi-turn workflow remains largely underexplored. To bridge this gap, we present MTPaperBananaBench, a benchmark for multi-turn diagram generation containing 292 images annotated with 3,518 user requirements. To reduce expensive human studies and enable scalable benchmarking, we construct a user simulator that, at each turn, identifies unsatisfied requirements and converts k of them into natural language feedback. Evaluating both requirement satisfaction and overall diagram quality reveals two key failure modes shared across baseline multiturn systems: (1) quality drift, where diagram quality progressively declines over turns, and (2) forgetting, where previously implemented features are lost in subsequent turns. To address these issues, we introduce PaperBanana-Interact, a multi-agent system that refines diagrams via an internal critique-and-refine loop. PaperBanana-Interact consistently improves rather than degrades diagram quality across turns, outperforming baselines by 11.9-18.6 points in quality score and reducing forgetting by 3.7-6.2 points.
Paper Metadata
Title: PaperBanana-Interact: Scientific Diagram Refinement with Multi-Turn Human Feedback
Authors: Xueqing Wu, Ashwin Balasubramanian, Bingxuan Li, Dawei Zhu, Kai-Wei Chang, Yale Song, Yiwen Song, Rui Meng, Tomas Pfister, Nanyun Peng (UCLA, Google, Peking University, UIUC)
Links: arXiv:2608.30241 (v1, 2026-08-31, cs.CL / cs.CV) · project page · code · MTPaperBananaBench dataset · interactive demo
Code status: released under Apache License 2.0 (file headers carry Copyright 2026 Google LLC). Two entry points ship in the repo: app.py is a Gradio demo that runs on your own Gemini API key, and main_multiturn.py is the benchmark driver. The command that reproduces the headline numbers is python main_multiturn.py --refiner pb_interact --refiner_internal_critic_rounds 10. Besides the three refiners proposed or baselined here (pb_interact.py, pb_directrefine.py, generative_refiner.py), agents/ also carries the full PaperBanana stack (planner, retriever, stylist, polish, visualizer) that produces the initial draft.
In one sentence: Scientific figures are never finished in one pass. This paper turns 3,518 hidden design preferences extracted from 292 author-drawn figures into a multi-turn benchmark, then shows that an agent loop with a history summarizer and a multi-objective critic makes figures better with every turn instead of worse: quality 50.3 to 61.2, forgetting 18.8% down to 12.6%.
Background and Motivation
Automating scientific figures has moved fast over the past year. AutoFigure drafts diagrams in code and lets a generative model polish them; Crafter and PaperBanana render the final image with a multimodal model and use an LLM only to optimize the prompt. Every one of these systems is single-turn: a method section plus a caption goes in, one figure comes out, done. Before building anything, the authors tested that assumption with 14 researchers who all had substantial academic writing and scientific illustration experience.
The assumption did not survive contact with the participants. All 14 asked for further revisions after seeing the initial draft. Twelve of them (86%) rated the refined figure higher than the draft, moving the average satisfaction score from 3.2 to 4.2, with per-person gains between 0.5 and 2 points. The more revealing number came out of the follow-up interviews: nine participants said the draft itself shaped or changed their idea of what the figure should look like, and in 64% of the sessions the refinement requests only appeared after the participant had seen a draft or an intermediate version. Author preferences are not a requirements document that can be written up front. They surface during the interaction.
The interviews also catalogued three recurring failures of the baseline system PaperBanana-DirectRefine: reduced factual faithfulness (29% of sessions), degraded aesthetic quality (43%), and failure to preserve requests that had already been satisfied (29%). The third one is the sharpest, because it is not a comprehension failure. The system understood the current request and destroyed a previous result while implementing it. The paper later gives the two failure modes names and measurements: quality drift and forgetting.
Why do models that are perfectly capable in a single turn fall apart over five turns? The failure analysis traces a clean causal chain. When a generative model refines the image directly, every turn is a full repaint with no explicit constraint on what must not move, so five rounds of re-layout wash the original design intent away: quality drops from 50.3 to 19.0. When an agent rewrites the image prompt instead, the drop is gentler (50.3 to 47.1) but takes a different shape: the new request gets bolted onto the layout, leaving unbalanced composition and large empty regions, plus stylistic drift nobody asked for. Both routes are missing the same thing, a cross-turn ledger of constraints that every later step can actually consult.
These baselines are not weak at the single-turn level. DirectRefine satisfies 68.6% of the requests raised in the turn they are raised, and NanoBananaPro reaches 56.2%. So the deficit is not instruction following, it is multi-turn state management. That is exactly why the overall requirement satisfaction ends up far below the per-turn number: GPT-Image-2 scores 67.4 per turn but only 52.9 overall, and Crafter refined by NanoBananaPro scores 56.2 per turn against 39.4 overall. The gap between those two columns is what later turns erased.
Figure 1 (paper teaser): diagram generators rarely satisfy an author's visual preferences in one turn, which forces multi-turn refinement with human feedback. MTPaperBananaBench measures this task, and PaperBanana-Interact suppresses the two baseline failure modes. The teaser itself was generated by PaperBanana-Interact.
Task Formulation and the Benchmark
In the single-turn setting the goal is a diagram $I_{0}$ from a source context $x_{0}$, which is the paper content plus a communicative intent that fixes the scope and focus of the figure. The multi-turn setting hands the system natural-language feedback $x_{t}$ at each turn $t\geq 1$ and asks for an updated diagram $I_{t}$ conditioned on the whole interaction history $[\mathbf{I}_{<t},\mathbf{x}_{\leq t}]$, where $\mathbf{x}_{\leq t}=[x_{0},\ldots,x_{t}]$ and $\mathbf{I}_{<t}=[I_{0},\ldots,I_{t-1}]$. The formalism is unremarkable on its own, but it puts one fact on the table: the history is a sequence of images, not a sequence of text. Every design decision that follows is a response to that.
Evaluating the task needs multi-turn interaction data, and real human-machine sessions are expensive and impossible to standardize across evaluation runs. The authors build a user simulator instead, driven by a pool of pre-annotated preferences that stay hidden from the generation system. The resulting benchmark holds 292 diagrams and 3,518 annotated requirements, about 12 per diagram.
The annotation strategy is the clever part. Instead of asking people to imagine what a user might request, the annotators treat the original author's reference figure $I_{H}$ as evidence and extract the salient design decisions it embodies as requirement statements. The pipeline has two stages: Gemini-3.1-Pro generates candidate requirements with taxonomy labels, then three human annotators independently review them. How much did the review change? 0.5% of candidates were removed and 24.7% were revised, so roughly one in four machine candidates needed a human edit. Inter-annotator agreement on a 207-requirement subset is Cohen's $\kappa=0.767$. To stop captions from leaking hidden requirements, Gemini-3.1-Pro also rewrites each caption to drop content redundant with the annotations, and both Gemini-3.1-Pro and Claude-Opus-4.8 independently check every requirement for faithfulness and label correctness, with disagreements flagged for closer inspection.
Requirements live in a three-by-three taxonomy: semantic category (content, organization, visual representation) crossed with impact severity (critical, communicative, stylistic). Annotators were told to keep the three semantic categories roughly balanced and to cover all three severity levels. That structure is what makes the later failure analysis decomposable, for instance it exposes that DirectRefine's drift concentrates in conciseness and readability.
| Axis | Value | Meaning (paper Appendix B) |
|---|---|---|
| Semantic category | Content | Which information appears, what is omitted, which functional relationships are shown, and the level of mathematical detail |
| Organization | How information is spatially arranged, ordered, and grouped | |
| Visual representation | Visual encodings, styles, metaphors, and overall aesthetic treatment | |
| Impact severity | Critical | A violation makes the figure scientifically, mathematically, or structurally incorrect |
| Communicative | Still technically correct, but the figure becomes confusing, overwhelming, or prone to misinterpretation | |
| Stylistic | Neither accuracy nor comprehension is affected; only aesthetic quality and professional polish drop |
Table 1: the MTPaperBananaBench requirement taxonomy (3 semantic categories by 3 severity levels).
The simulator works as follows. The requirement list $R=[r_{1},\ldots,r_{N}]$ is hidden. At turn $t$ an LLM-as-judge evaluator $\mathcal{E}$ checks each $r_{i}\in R$ against the latest diagram $I_{t-1}$, producing the failed set $F_{t-1}$. If it is non-empty, the simulator takes the first $k$ failures in a predefined order ($k$ is a hyperparameter; the paper reports $k=1$ and $k=3$) and prompts a vision-language model to phrase them as one natural user utterance $x_{t}$. The evaluator's rationales for why each requirement failed are handed to the simulator as well, so the complaints read like a real person reacting to a real figure. Interaction stops when $F_{t-1}=\varnothing$ or the turn budget $T_{\max}=5$ is reached. Requirement order is randomly permuted once after annotation and then fixed, which removes ordering bias across runs.
Figure 2 (paper Figure 3): the annotation pipeline on top (LLM candidates plus human review), and the inference-time user simulator below. Each turn evaluates the current diagram against every requirement, extracts $k$ failures, and turns them into natural-language feedback.
Method: Two Baselines and PaperBanana-Interact
Baseline one, a generative model as the refiner. The cheapest option is to feed the previous image plus all user utterances so far to an image generation model $\mathcal{V}$ and let it repaint:
$$I_{t}=\mathcal{V}(I_{t-1},\mathbf{x}_{\leq t}).$$
The experiments make the cost of this route explicit. NanoBananaPro refining a PaperBanana draft for five turns takes quality from 50.3 to 19.0; starting from a Crafter draft it is worse, 35.4 down to 2.9. Every repaint is an unconstrained re-layout, and after five of them the original design intent is effectively gone.
Baseline two, PaperBanana-DirectRefine. Agentic diagram systems keep a detailed image-generation prompt $P$ produced by a pipeline $\mathcal{A}$ and rendered by a visualizer $\mathcal{V}$. The first turn is $P_{0}=\mathcal{A}(x_{0})$, $I_{0}=\mathcal{V}(P_{0})$. The multi-turn extension factors out the prompt rewriting step as $\mathcal{A}_{r}$:
$$P_{t}=\mathcal{A}_{r}(P_{t-1},\mathbf{I}_{<t},\mathbf{x}_{\leq t}),\quad I_{t}=\mathcal{V}(P_{t}).$$
The authors instantiate $\mathcal{A}_{r}$ as a direct prompt to a vision-language model $\mathcal{M}_{\text{vlm}}$. To control compute and avoid long-context degradation, the model gets the full sequence of user utterances but only the most recent image and prompt:
$$P_{t}=\mathcal{M}_{\text{vlm}}(I_{t-1},P_{t-1},\mathbf{x}_{\leq t}).$$
That truncation is the crux. Without the historical images the model has to guess from text what earlier edits actually looked like, so previously satisfied requirements get overwritten during later rewrites. This is the mechanism behind forgetting. Quality retention is genuinely better than repainting (50.3 to 47.1), but the distance between 68.6% per-turn satisfaction and an 18.8% forgetting rate shows the system cannot hold cross-turn state.
PaperBanana-Interact, part one: the summarizer $\mathcal{S}$. Making every agent re-process a multi-image history is both expensive and distracting, and successive revisions are visually redundant enough to compress hard. So a summarizer distills the history into a textual memory shared by all downstream agents:
$$M_{t}=\mathcal{S}(\mathbf{I}_{<t},\mathbf{x}_{\leq t}).$$
$M_{t}$ is not a generic conversation summary. It records three specific things: what each diagram depicts, how it changed relative to the previous revision, and whether those changes actually addressed the user's request in that turn.
Part two: the critic-refiner inner loop. Turn $t$ starts from the previous turn's artifacts, $I_{t}^{(0)}=I_{t-1}$ and $P_{t}^{(0)}=P_{t-1}$. At inner iteration $\tau$ a critic $\mathcal{C}$ inspects the current version and produces a critique:
$$c_{t}^{(\tau)}=\mathcal{C}(I_{t}^{(\tau-1)},\mathbf{x}_{\leq t},M_{t}).$$
If the critic decides more revision is needed, a refiner $\mathcal{R}$ folds that critique into an updated prompt:
$$P_{t}^{(\tau)}=\mathcal{R}(I_{t}^{(\tau-1)},P_{t}^{(\tau-1)},c_{t}^{(\tau)},\mathbf{x}_{\leq t},M_{t}).$$
The visualizer then renders it, $I_{t}^{(\tau)}=\mathcal{V}(P_{t}^{(\tau)})$, and hands the result back to the critic. The loop terminates when the critic sees nothing left to fix or when the iteration budget $\tau_{\max}=10$ runs out. Note the two time scales: the outer scale is the user turn $t$ (at most 5), the inner scale is self-iteration $\tau$ under a single piece of feedback (at most 10). The user says one sentence; the system may revise ten times behind it.
Part three: the multi-objective critic. This is the core design, and the only component in the ablation that moves all four metrics at once. Instead of one overall verdict, the critic must emit structured JSON with a separate critique entry per constraint: the current user request, each prior user request, faithfulness to the source context (method section and caption), and overall presentation quality (layout, readability, legend). Giving every prior turn its own field is aimed straight at forgetting, because a new edit that quietly sacrifices an old one now has to leave a line in the critic's ledger.
Figure 3 (paper Figure 4): system overview. A summarizer first compresses the multi-image history into compact memory, then a refinement loop runs in which the multi-objective critic evaluates the diagram against each constraint and drives the prompt update and re-render.
flowchart TD SIM["User simulator
scores the current diagram per requirement
takes the first k failures"] --> UT["Feedback of this turn x_t"] UT --> SUM["Summarizer S
compresses multi-image history into memory M_t
what each diagram shows and whether it landed"] SUM --> CRIT["Multi-objective critic C
structured JSON output"] CRIT --> K1["Current turn request"] CRIT --> K2["Every prior turn request one field each"] CRIT --> K3["Faithfulness to method section and caption"] CRIT --> K4["Presentation quality layout legend"] K1 --> GATE{"Any edit still needed"} K2 --> GATE K3 --> GATE K4 --> GATE GATE -->|no| KEEP["Keep current diagram and close the turn"] GATE -->|yes| REF["Refiner R
rewrites the image generation prompt P_t"] REF --> VIS["Visualizer V
renders the new diagram I_t"] VIS --> BUD{"tau_max = 10 reached"} BUD -->|no| CRIT BUD -->|yes| KEEP KEEP --> NEXT["Advance to turn t+1
memory updated with the new history"] NEXT --> SIM
Figure 4: the two nested loops. The outer loop is the user turn (at most $T_{\max}=5$); the inner loop is critique, refine, and render under one piece of feedback (at most $\tau_{\max}=10$), exiting early as soon as the critic sees nothing left to fix.
How the Metrics Are Actually Computed
Overall requirement satisfaction averages the binary verdicts of all $N$ requirements on the final diagram $I_{T}$, $\text{Req}=\frac{1}{N}\sum_{i=1}^{N}\mathbf{1}[\mathcal{E}(r_{i},I_{T})=\text{satisfied}]$. Per-turn satisfaction $\text{Req}^{\text{pt}}$ looks only at the requirements raised in a given turn and asks whether they were met within that same turn, which isolates single-turn instruction following. Forgetting@any is pooled over (turn, datapoint, requirement) events:
$$\text{Forget@any}=\frac{\sum_{t=1}^{T-1}\sum_{dp}\mathbf{1}[\,r\ \text{surfaced and satisfied at turn}\ t\ \wedge\ \text{unsatisfied after a later real edit}\,]}{\sum_{t=1}^{T-1}\sum_{dp}\mathbf{1}[\,r\ \text{surfaced and satisfied at turn}\ t\ \wedge\ \text{a later real edit exists}\,]}.$$
The denominator condition is not decorative. If every later turn stopped early because all requirements were met, forgetting was never possible, so those events must be dropped or the rate is systematically understated. The final turn is likewise excluded as a surfacing turn, since nothing after it can break anything.
Diagram quality reuses the PaperBanana pairwise protocol: the generated diagram $I_{T}$ is compared against the human reference $I_{H}$ on four dimensions, faithfulness, conciseness, readability, and aesthetics. Each dimension yields a win, tie, or loss, and a two-tier vote aggregates them into an overall outcome scored 100 for a win, 50 for a tie, and 0 for a loss. The paper states that faithfulness and conciseness are the primary dimensions and readability and aesthetics the secondary ones: a decisive primary result (one diagram wins both, or wins one and ties the other) settles the outcome, and only a primary tie falls through to the secondary tier.
Judge reliability is checked three ways. A human evaluator independently judged 200 requirements and reached Cohen's $\kappa=0.812$ with the VLM judge. Swapping in Claude-Opus-4.8 as the requirement judge on the same DirectRefine outputs ($k=3$, five turns) gives $\kappa=0.663$ against Gemini-3.1-Pro. And a human annotator ranked three system outputs per sample with system identities hidden across 150 datapoints, preferring PaperBanana-Interact over NanoBananaPro in 81.3% of cases and over DirectRefine in 76.7%. Together these put the self-preference concern in a discussable rather than fatal range.
Experimental Results
All agentic systems use Gemini-3.1-Pro as the language backbone and NanoBananaPro as the image backbone, Interact runs with $\tau_{\max}=10$, and the simulator runs up to $T_{\max}=5$ turns. AutoFigure and AutoFigure-Edit are excluded from the multi-turn evaluation because their single-turn quality is 1.9 and 0.3, and there is nothing meaningful to refine from a floor that low.
| Single-turn generator | Multi-turn refiner | Qual ↑ | Req ↑ | Reqpt ↑ | Forget ↓ |
|---|---|---|---|---|---|
| AutoFigure | - | 1.9 | 20.4 | - | - |
| AutoFigure-Edit | - | 0.3 | 16.5 | - | - |
| NanoBananaPro | - (single turn) | 22.4 | 18.1 | - | - |
| NanoBananaPro | NanoBananaPro | 12.5 | 39.4 | 53.8 | 20.9 |
| GPT-Image-2 | - (single turn) | 41.6 | 25.0 | - | - |
| GPT-Image-2 | GPT-Image-2 | 25.7 | 52.9 | 67.4 | 15.5 |
| Crafter | - (single turn) | 35.4 | 18.4 | - | - |
| Crafter | NanoBananaPro | 6.2 | 39.4 | 56.2 | 19.2 |
| Crafter | DirectRefine | 30.5 | 47.6 | 68.1 | 15.0 |
| Crafter | Interact | 48.1 | 52.1 | 73.4 | 10.3 |
| PaperBanana | - (single turn) | 50.3 | 25.2 | - | - |
| PaperBanana | NanoBananaPro | 19.0 | 45.2 | 56.2 | 19.0 |
| PaperBanana | DirectRefine | 47.1 | 52.6 | 68.6 | 18.8 |
| PaperBanana | Interact | 61.2 | 58.0 | 77.2 | 12.6 |
Table 2: main results with the $k=1$ user simulator (paper Table 1). Qual is the quality score against the human reference, Req is overall requirement satisfaction on the final diagram, Reqpt is per-turn satisfaction, and Forget is the forgetting rate.
Read down the $k=1$ table and three things stand out. First, multi-turn interaction does buy requirement satisfaction: PaperBanana alone reaches 25.2, and any refiner pushes it past 45, simply because hidden requirements are released turn by turn. Second, satisfaction gains do not automatically bring quality gains. With NanoBananaPro as the refiner, Req climbs from 25.2 to 45.2 while Qual collapses from 50.3 to 19.0, which is what it looks like when a system buys compliance with the picture. Third, the 10 to 20 point gap between per-turn and overall satisfaction is forgetting, and it sits between 14.1% and 22.9% for every baseline, against 12.6% for Interact.
Interact is the only refiner whose quality ends above its single-turn starting point: 50.3 in, 61.2 out after five turns. That is not noise. Across five reruns the mean and standard deviation are 61.2 ± 1.8 for quality, 58.0 ± 0.4 for Req, 77.2 ± 0.3 for Reqpt, and 12.6 ± 1.2 for forgetting. The same direction holds from a weaker start, Crafter at 35.4 refined to 48.1. The abstract's claim of 11.9 to 18.6 quality points over baselines and 3.7 to 6.2 points less forgetting comes from these rows.
| Single-turn generator | Multi-turn refiner | Qual ↑ | Req ↑ | Reqpt ↑ | Forget ↓ |
|---|---|---|---|---|---|
| NanoBananaPro | - (single turn) | 22.4 | 18.1 | - | - |
| NanoBananaPro | NanoBananaPro | 8.6 | 76.5 | 59.6 | 21.5 |
| GPT-Image-2 | - (single turn) | 41.6 | 25.0 | - | - |
| GPT-Image-2 | GPT-Image-2 | 22.4 | 88.5 | 71.0 | 14.1 |
| Crafter | - (single turn) | 35.4 | 18.4 | - | - |
| Crafter | NanoBananaPro | 2.9 | 74.6 | 59.1 | 21.6 |
| Crafter | DirectRefine | 23.5 | 87.8 | 72.2 | 16.0 |
| Crafter | Interact | 42.1 | 93.4 | 79.0 | 12.3 |
| PaperBanana | NanoBananaPro | 18.3 | 76.9 | 59.8 | 22.9 |
| PaperBanana | DirectRefine | 42.6 | 87.6 | 71.0 | 19.1 |
| PaperBanana | Interact | 54.5 | 94.2 | 79.6 | 13.0 |
Table 3: main results with the $k=3$ user simulator (paper Table 2). Three requirements are released per turn, so feedback is longer and constraints interact.
The $k=3$ setting raises the difficulty: three requests at once, longer utterances, and requirements that pull against each other. Overall satisfaction rises accordingly, since more requirements get released across five turns, and Interact reaches 94.2 from a PaperBanana start. Quality drift, however, is amplified. DirectRefine falls from 47.1 at $k=1$ to 42.6, NanoBananaPro to 18.3, and Crafter refined by NanoBananaPro ends at 2.9. Interact's own gain is more modest here (54.5 against a 50.3 start), which the authors acknowledge, but the sign is still upward rather than downward.
Figure 5 (paper Figure 5): refiners iterating on PaperBanana outputs for five turns with the $k=3$ simulator. Top, quality per refinement turn; bottom, the final per-dimension quality breakdown. NanoBananaPro drifts on all four dimensions, while DirectRefine loses ground mainly in conciseness and readability.
The per-dimension breakdown separates the two failure modes. NanoBananaPro degrades everywhere, with sudden drops in aesthetics and distorted rendering artifacts as the visible symptoms. DirectRefine keeps most of the picture intact but integrates user requests awkwardly, producing unbalanced layouts, large empty areas, and unexplained stylistic drift, which shows up as losses in conciseness and readability. Interact differs mechanically in exactly the place that matters: multiple internal iterations inside one turn let it fix layout and styling on the spot instead of carrying unresolved constraints into the next turn.
| Critic | History | $\tau_{\max}$ | Qual ↑ | Req ↑ | Reqpt ↑ | Forget ↓ |
|---|---|---|---|---|---|---|
| MO multi-objective | Sum memory | 10 | 61.2 | 58.0 | 77.2 | 12.6 |
| Naive unstructured | Sum | 10 | 58.9 | 57.1 | 72.3 | 13.1 |
| MO | None, only $I_{t-1}$ | 10 | 58.0 | 58.1 | 77.1 | 14.7 |
| MO | Full multi-image history | 10 | 58.9 | 57.6 | 73.5 | 11.6 |
| MO | Sum | 5 | 60.6 | 57.4 | 75.5 | 13.4 |
| MO | Sum | 3 | 53.3 | 55.9 | 74.1 | 13.8 |
| MO | Sum | 1 | 43.5 | 54.2 | 72.3 | 16.8 |
Table 4: ablations (paper Table 3). All runs refine PaperBanana outputs for five turns with the $k=1$ simulator. MO is the multi-objective critic, Sum the summarized memory, None no history beyond $I_{t-1}$, and Full the complete multi-image history.
The three history rows are the most informative part of the ablation. With no history (None), forgetting rises from 12.6 to 14.7 and quality drops 3.2 points, while overall satisfaction barely moves (58.1 against 58.0). So what the memory buys is "do not lose earlier edits", not "satisfy more new requirements". Full history gives the lowest forgetting (11.6) at the price of per-turn satisfaction falling from 77.2 to 73.5, final quality down to 58.9, and 37.1% more input tokens. The compressed textual memory wins both quality and per-turn satisfaction outright, which is the real argument for the summarizer: it is not a cost-saving compromise, it is the best-performing setting.
The iteration budget has the steepest effect. From $\tau_{\max}=10$ to 5 to 3 to 1, quality goes 61.2, 60.6, 53.3, 43.5, and forgetting worsens monotonically from 12.6% to 13.4%, 13.8%, and 16.8%. At $\tau_{\max}=1$, meaning a single refinement pass with no further iteration, quality is 43.5, below the 50.3 single-turn starting point. Remove the inner loop and the system degenerates into a more verbose DirectRefine. Swapping the multi-objective critic for a naive unstructured one costs 4.9 points of per-turn satisfaction and 2.3 points of quality with slightly higher forgetting, so the requirement to produce one verdict per constraint is doing real work beyond prompt wording.
Figure 6 (paper Figure 6): qualitative quality drift. DirectRefine integrates the user request awkwardly and leaves an unbalanced layout with excessive empty space, while Interact iteratively refines layout and styling and lands a higher-quality figure.
Figure 7 (paper Figure 7): qualitative forgetting. DirectRefine implements the requested change at first, then loses it during later turns that repeatedly restyle the same blocks.
The bill matters too. Generating an initial diagram with PaperBanana and refining it with Interact at $\tau_{\max}=10$ for five turns costs 186.8k input and 6.2k output tokens for Gemini-3.1-Pro plus 5.7k and 5.3k for NanoBananaPro in the first round, then 159.4k and 11.2k plus 7.1k and 5.0k per subsequent refinement turn. Most of that overhead comes from the internal iterations. The distribution of iteration counts concentrates at small values and decays, but there is a 22% spike pinned at the maximum, meaning roughly one turn in five never satisfies the stopping criterion within budget.
Where the Method Lives in the Code
The released implementation is more specific than the paper text, and a few decisions are only visible in the source.
| Paper component | Code location | Detail only visible in the source |
|---|---|---|
| Summarizer $\mathcal{S}$ | STEP 0 and CONTEXT_SYSTEM_PROMPT in agents/refiner/pb_interact.py | The memory is not free text but structured JSON keyed by turn: one context_diagram{n} per figure and one context_user_turn{n} per user utterance. The prompt casts the summarizer as a lead visual designer acting as note-taker for a diagram revision conversation, and demands each note be self-contained, so a reader who never saw the images can reconstruct what was shown, what the user objected to, and why. It runs once at $\tau=0$ (j == 0 in the code) and is reused by later internal iterations. |
| Multi-objective critic $\mathcal{C}$ | STEP 1 in the same file, schema assembled dynamically | The four critique routes map to four JSON keys: critique_latest_turn, one critique_prior_turn{n} per actual prior turn, critique_source_content, and critique_presentation. The critique step deliberately withholds the current prompt from the model so it cannot rationalize the existing description; the prompt is injected only at STEP 3, when rewriting actually happens. |
| Loop termination | utils/paperviz_processor.py::process_single_feedback | The processor owns the inner loop and the round index [i, j], where $i$ is the user turn and $j$ the internal iteration. STEP 2 is a gating question: if the model answers "No", the agent writes No changes needed. and returns without touching the description, which avoids a pointless re-render. If rendering fails, the last good image is kept and the loop breaks instead of feeding a broken image to the next critique. |
| User simulator | utils/user_simulator_multi_req.py::simulate_user_feedback | It walks req_order and takes the first n_surfaced_req_per_turn unsatisfied requirements; if there are none it sets early_stopped=True and returns, which is exactly the $F_{t-1}=\varnothing$ stop. The chat history entry stores the utterance plus surfaced_index and each surfaced requirement's evaluation reasoning, which the metric code reuses later. |
| Forgetting and per-turn satisfaction | main_multiturn.py::_forgetting_rate and _surfaced_addressed_ratio | Events are pooled over (turn, datapoint, requirement), so the same requirement surfaced on two turns counts twice. Early-stopped turns are skipped on both sides of the ratio because they never happened, and their trailing surfaced_index is a stale copy of the last real turn's request. |
Table 5: mapping from paper components to the open-source implementation (shirley-wu/PaperBanana-Interact).
One place where the paper and the code disagree is worth flagging before anyone tries to reproduce the quality column. Appendix F.1 states that faithfulness and conciseness are the primary dimensions with readability and aesthetics secondary. In utils/eval_toolkits.py, however, the first voting tier uses faithfulness and readability, and only a tie there falls through to conciseness and aesthetics (_determine_tier_outcome(faithfulness, readability), then _determine_tier_outcome(conciseness, aesthetics)). The two orderings disagree on combinations such as faithfulness win, conciseness loss, readability tie. The same function also carries a fallback rule that awards the win to the human reference whenever a turn fails to render an image, which helps explain extreme lows such as 2.9 for NanoBananaPro on a Crafter start.
Configuration defaults are another trap. utils/config.py ships max_critic_rounds=3, refiner_internal_critic_rounds=1, n_surfaced_req_per_turn=1, and gemini-3.1-flash-image-preview as the default image model, whereas the reported numbers use $T_{\max}=5$, $\tau_{\max}=10$, and NanoBananaPro (gemini-3-pro-image), all supplied via command-line overrides. A run with stock defaults reproduces the $\tau_{\max}=1$ ablation row (quality 43.5), not the headline result.
Limitations
Stated by the authors: visual hallucination and an assistive-only role. Appendix A acknowledges the ethical risk of generative models, in particular plausible-looking but factually incorrect representations of scientific content. The mitigation is positioning: the tool is an assistive system only, with a prominent disclaimer in the README and on the demo landing page, and a planned license requiring any academic publication that uses its images to disclose that they were generated by the system and to include a section discussing the role of AI in the research. The authors stress that users must not over-rely on such systems and must keep rigorous human oversight to preserve the accuracy and integrity of published illustrations.
Stated by the authors: forgetting is reduced, not eliminated. Interact still forgets 10% to 13% of previously satisfied requirements, so roughly one in every eight to ten earlier requests is lost during later edits. For a tool aimed at camera-ready figures, that number alone makes manual review non-optional.
Stated by the authors: single-run numbers under compute constraints. Appendix D says plainly that most main-text numbers come from a single run because of computational limits, with five reruns reported only for the main setting. Combined with the 22% of turns that stop at the $\tau_{\max}$ ceiling, the true headroom on hard examples is unmeasured: a larger budget might buy more quality, or might only buy more tokens.
Additional concern one: Gemini is simultaneously the player, the referee, and the user. Gemini-3.1-Pro is the language backbone of the system, the judge for both quality and requirement satisfaction, and the model that phrases the simulated user's complaints. The paper hedges with a Claude-Opus-4.8 cross-judge ($\kappa=0.663$) and a 150-sample human ranking, but no equivalent check exists on the simulator side. If the simulated complaints are phrased in a style that Gemini-family refiners digest more easily, the relative ranking of systems could shift.
Additional concern two: the requirement pool is anchored to the original authors' figures. Deriving requirements from the design decisions embodied in $I_{H}$ guarantees that entries are expert and decidable, but it also installs one author's preferences as the only correct answer. Figure design routinely admits several equally valid solutions (horizontal versus vertical stacking, whether to expand the mathematical detail), and a binary satisfaction verdict scores such divergence as failure. The quality metric has the same character, since it is a win rate against a reference rather than an absolute measure. That is also why Interact tops out at 61.2: it remains a substantial distance from the human reference.
Additional concern three: a 14-person formative study and retrospective self-rating. The motivating study has 14 participants, and the human ranking plus the 200-requirement manual judgement were each performed by a single annotator whose independence from the evaluated systems is not stated. The 86% improvement figure is impressive but retrospective: participants rated the final figure and then looked back at the draft, a design that naturally amplifies perceived improvement.
Additional concern four: unit cost and domain scope. Roughly 160k input tokens per refinement turn, over five turns per figure, is a hard constraint for any workflow that needs figures in bulk. The method and the benchmark are also tied to scientific diagrams: faithfulness is judged against a source context (method section plus caption), so in domains without an explicit source text, such as illustration or product design, the third critique route loses its footing.
Summary and Outlook
The contribution here is not a single clever trick but the completion of a setting that had been skipped: a task formalization, a reproducible multi-turn benchmark (292 diagrams, 3,518 requirements, a three-by-three taxonomy, annotation agreement $\kappa=0.767$), two failure modes turned into measurements (quality drift and forgetting), and a system that suppresses both. Two design choices carry the effect: compressing multi-image history into turn-indexed structured textual memory, and forcing the critic to emit one verdict per constraint. The first is the best-performing history setting in the ablation rather than merely the cheapest, and the second writes "a new edit may not silently sacrifice an old one" into the architecture.
For anyone building agent systems, the transferable lessons are concrete. Do not expect the model to remember cross-turn state; keep an explicit constraint ledger that every step can read. Do not show the critic the intermediate artifact it is about to rewrite, or critique decays into justification of the current draft. And treat the inner-loop budget as a real performance dial, since $\tau_{\max}$ from 1 to 10 is worth 17.7 quality points, not as optional polish. The open problems are equally clear: driving the residual 10% to 13% forgetting toward zero, actually solving the 22% of turns that hit the iteration ceiling, and separating the referee from the player.
Golden quote: "Figure design is naturally iterative: reviewing an initial draft often reveals refinement opportunities that are difficult to anticipate upfront." This is the load-bearing claim of the whole paper. If preferences only form after the author sees a figure, no single-pass generator can be sufficient, and the 64% of sessions in which requests emerged post-draft is the empirical backing.
