MEMENTO: Memory-Guided Memetic Code-as-Policy Evolution

MEMENTO: Memory-Guided Memetic Code-as-Policy Evolution

Paper: MEMENTO: Memory-Guided Memetic Code-as-Policy Evolution
Authors: Alkis Sygkounas, Victor Aregbede, Amy Loutfi, Andreas Persson (Örebro University AASS)
Link: arXiv:2607.22832　|　Code: github.com/sygkounas/MEMENTO　|　Platform: Franka Panda / AI2-THOR

One-line summary: A single-elite memetic framework evolves code-as-policy by first evolving a rollout evaluator, then improving policy programs via memory-guided hill-climbing, macro-mutation, and crossover—achieving 97-100% success on Tower-of-Hanoi and AI2-THOR with sim-to-real transfer.

Background and Motivation

Long-horizon embodied tasks require policies to execute many dependent actions before task success can be observed. Earlier actions determine the task state encountered by later actions—correct action order is insufficient; each action must also be physically executed from the observed state. This creates a long-horizon credit-assignment problem: success is observed only after many dependent actions. Existing methods mitigate delayed feedback through external structure (demonstrations, hierarchy, reusable skills) or experience generation/crediting changes (exploration, relabeling, temporal abstraction, return redistribution), but don't directly expose the decision logic of a failed policy or identify which part should be revised.

When policies are represented as executable programs (code-as-policy), decision logic is written directly as code and can be inspected and revised. LLMs generate robot policy code that processes perception outputs and parameterizes control-primitive APIs. But in long-horizon tasks, a generated program can execute without errors yet fail at control logic. Failure feedback has been used to characterize policy errors and guide refinement, formulating policy improvement as execution-guided program search. But binary task success alone is too sparse to guide revision of long-horizon policy programs.

MEMENTO's core idea: use an LLM-driven memetic algorithm for code-as-policy evolution, first evolving a rollout evaluator that maps policy rollouts to scalar fitness and structured feedback metrics, then improving policies via memory-guided hill-climbing, macro-mutation, and crossover.

Method

1. Evaluator Evolution

The evaluator $E$ receives a task rollout $\mathcal{T}$, outputting fitness $F\in[0,1]$ and feedback metrics $\mathbf{M}$. The evaluator itself evolves via LLM-driven macro-mutation: starting from $n$ initial evaluators $\{E_0^{(j)}\}$, each round executes rollouts producing $(F,\mathbf{M})$, selects the best evaluator via majority vote $\mathcal{V}_k$ over $k$ independent LLM judgments, then generates the next generation via macro-mutation $q_{\text{eval}}^{\text{macro}}$. Evaluator evolution is independent of policy search and fixed before policy evolution begins.

2. Single-Elite Memetic Policy Evolution

After $E^*$ is fixed, policy search proceeds from a single elite candidate state $\mathcal{C}_g=\mathcal{C}(p_g)$. Each generation explores three branches:

Hill-climb branch: Maintains a current accepted candidate and rejected-proposal memory $\mathcal{H}$. At each step, samples a new policy from the accepted state and rejection memory:

$$p_{g,h}^{\text{new}}\sim q_{\text{hc}}\!\left(\mathcal{C}_{g,h-1}^{\text{hc}},\mathcal{H}_{g,h-1}\right)$$

Acceptance requires non-decreasing fitness:

$$\mathcal{C}_{g,h}^{\text{hc}}=\begin{cases}\mathcal{C}_{g,h}^{\text{new}},&\text{if }F(\mathcal{C}_{g,h}^{\text{new}})\geq F(\mathcal{C}_{g,h-1}^{\text{hc}})\\[2pt]\mathcal{C}_{g,h-1}^{\text{hc}},&\text{otherwise}\end{cases}$$

Rejections append edit summaries to $\mathcal{H}$, avoiding repeated attempts of the same failed modification. This is the core of "memory guidance"—rejection memory tells the LLM which modifications have proven ineffective.

Macro-mutation branch: Samples $K_{\text{macro}}$ independent mutations from the generation elite, taking the highest-fitness:

$$\mathcal{C}_{g}^{\text{macro}}=\arg\max_{\mathcal{C}\in\{\mathcal{C}_g,\mathcal{C}_{g,1}^{\text{macro}},\ldots\}}F(\mathcal{C})$$

Crossover branch: Recombines $K_{\text{cross}}$ candidates from hill-climb and macro-mutation outputs:

$$p_{g,r}^{\text{cross}}\sim q_{\text{cross}}\!\left(\mathcal{C}_{g}^{\text{hc}},\mathcal{C}_{g}^{\text{macro}}\right)$$

Elite selection: The highest-fitness branch output becomes the next generation elite:

$$\mathcal{C}_{g+1}=\arg\max_{\mathcal{C}\in\{\mathcal{C}_{g}^{\text{hc}},\mathcal{C}_{g}^{\text{macro}},\mathcal{C}_{g}^{\text{cross}}\}}F(\mathcal{C})$$

graph TD
  A[Initial Policy p0] --> B[Evaluator E* Evolution]
  B --> C[Fixed Evaluator]
  C --> D[Gen-g Elite C_g]
  D --> E[Hill-climb: Memory-guided micro-mutation]
  D --> F[Macro-mutation: Independent large mutation]
  E --> G[Crossover: Recombine best of both branches]
  F --> G
  G --> H[Elite Selection: Max fitness]
  H --> I[Gen g+1 Elite C_{g+1}]
  I --> E
  style E fill:#f5a623,stroke:#b97316,color:#fff
  style G fill:#4a90d9,stroke:#2c5f8a,color:#fff
  style I fill:#7ed321,stroke:#4a8a14,color:#fff

Experimental Results

Main Comparison

Evaluated on two long-horizon domains: Robosuite Franka Tower-of-Hanoi (4 cubes, 15-step optimal solution) and AI2-THOR household interaction (apple in microwave + bread in fridge). MEMENTO runs $G=5$ generations with $K_{\text{hc}}=10$, $K_{\text{macro}}=5$, $K_{\text{cross}}=5$.

Method	Robosuite Fitness	Robosuite Success	AI2-THOR Fitness	AI2-THOR Success
MEMENTO (full)	0.99±0.02	0.97±0.06	0.98±0.00	1.00±0.00
Hill-only	0.50±0.04	0	0.46±0.04	0
Without-Crossover	0.55±0.02	0	0.51±0.04	0
Eureka (baseline)	—	0	—	0
REvolve (baseline)	—	0	—	0

Figure 3: Main evolution results on Robosuite Tower-of-Hanoi and AI2-THOR. Solid curves show best-so-far fitness, hollow squares show success rate.

Generalization

Tested on held-out Robosuite object configurations and unseen AI2-THOR kitchens (FloorPlan4-29). MEMENTO maintains high success in both domains while baselines completely fail out-of-distribution. Single-scene policy evolution achieves only 0.09±0.02 cross-kitchen generalization (2/29, 3/29, 3/29), showing multi-scene search is necessary for cross-scene generalization.

Generalization Setting	Train Success	Held-out Success
Robosuite (held-out objects)	0.97	0.73
AI2-THOR (multi-scene train)	1.00	0.72
AI2-THOR (single-scene train)	1.00	0.09

Figure 2: Training task environments. Left: Robosuite four-cube Tower-of-Hanoi. Right: AI2-THOR household interaction scene.

Sim-to-Real Transfer

The best Robosuite policy is deployed on a physical Franka robot. During simulation search, the policy acquired a closed-loop execution structure—object pose filtering, pick-and-place target re-derivation, alignment checks, and post-placement retry. Physical deployment uses SAM3 segmentation for RGB-D object pose estimation. After 3 LLM-assisted calibration steps, 10 physical trials achieve 0.90 success rate; the single failure was due to perception error.

Figure 1: MEMENTO conceptual overview. Top row shows task environments; bottom row shows evaluator evolution and three-branch policy search.

Limitations

Author-stated: Hill-only and macro-mutation-only ablations fail to produce successful rollouts under the tested candidate budget, indicating the crossover branch is needed to combine partial executable strategies into task-completing policies. Single-scene policy evolution overfits to scene-specific object placements, navigation routes, and interaction viewpoints on AI2-THOR.

Analysis: 5 generations may be insufficient for more complex tasks. The evaluator itself is evolved by an LLM, whose quality is bounded by the LLM's understanding of rollouts—if the LLM misjudges rollout quality, the fitness signal misleads policy search. Sim-to-real requires 3 LLM-assisted calibration steps, indicating the sim-real gap still needs human intervention. The Tower-of-Hanoi symbolic solving order is fixed; the policy only adapts execution—for tasks where the decision logic itself must be discovered, the framework's effectiveness is unverified. Computationally, each generation requires $K_{\text{hc}}+K_{\text{macro}}+K_{\text{cross}}=20$ LLM proposals and rollout evaluations, totaling 100 over 5 generations—LLM API and simulation costs are not thoroughly reported.

Conclusion and Future Work

MEMENTO's core contribution elevates code-as-policy improvement from independent variant selection to memetic evolution with a sequential local improvement phase. First evolving the evaluator solves the sparsity of binary success signals—structured feedback metrics $\mathbf{M}$ guide which part of the policy logic the LLM should modify. The rejected-proposal memory $\mathcal{H}$ is the hill-climb branch's key innovation—telling the LLM which modifications have proven ineffective, enabling genuine "memory guidance" rather than random attempts. The three-branch search ablation proves that local refinement and macro-mutation alone cannot produce successful policies; crossover combines hill-climbing's precise progressive improvement with macro-mutation's exploratory power to assemble complete task-solving strategies. 97-100% success rates and 0.90 sim-to-real success validate the framework's effectiveness on long-horizon embodied tasks.

Evolving code-as-policy isn't just having the LLM write more versions—the key is letting the evaluator learn "what counts as doing well," then letting memory tell the search "which paths are dead ends."