Heretic: Rewriting Model Behavior Without Retraining

flowchart TD
  A["Load model and configuration"] --> B["Probe an efficient batch size"]
  B --> C["Detect response prefixes and thinking wrappers"]
  C --> D["Evaluate the unmodified baseline"]
  D --> E["Capture examples and intermediate model values"]
  E --> F["Sample modifier parameters with Optuna"]
  F --> G["Reset the model and apply one modifier"]
  G --> H["Generate responses and compute scorer values"]
  H --> I{"Trial budget complete?"}
  I -- "No" --> F
  I -- "Yes" --> J["Rank the Pareto frontier"]
  J --> K["Export adapter or merged model"]
  K --> L["Chat review and independent benchmarks"]
  

Arbitrary-Rank Ablation

ARA is the default modifier. It captures module inputs and outputs for 400 prompts from a harmless-behavior dataset and 400 from a harmful-behavior dataset, then inserts low-rank adapters with rank 50 into selected attention output projections and MLP down projections. The mathematical method is called arbitrary-rank ablation, but the implementation deliberately restricts itself to a low-rank approximation. The code notes that singular values tend to decay after a few dozen dimensions, and low-rank adapters are cheaper to optimize, export, and merge than a dense replacement matrix.

The objective is geometric rather than label-based. For each captured module, let g and b be the original outputs for good and bad prompts, and let g' and b' be their outputs after the edit. The preservation term is the mean squared displacement of the good outputs:

$$P = \operatorname{mean}\left((g' - g)^2\right)$$

The steering term compares bad outputs with local neighborhoods of the original good and bad outputs. Here D_k(a,b) is the mean Euclidean distance from each vector in a to its k nearest neighbors in b, and w_o controls overcorrection:

$$S = D_k(b', g) - w_o D_k(b', b)$$

Optuna chooses the final weights that combine preservation and steering:

$$L = w_p P + w_s S$$

This objective produces a local transfer rather than a single replacement answer. Good outputs should stay near their original positions. Bad outputs should move toward the good neighborhood and away from the bad one. The neighbor count controls how broad that local neighborhood is. If it is too small, the edit may chase a few peculiar examples. If it is too large, the neighborhood can become too diffuse to express the intended behavior boundary.

ARA searches six parameters per trial: the first and last edited layers, the preservation weight, the steering weight, the relative overcorrection weight, and the number of neighbors. The first layer is sampled from the beginning through the midpoint of the stack. The exclusive end layer is sampled after the midpoint, so the range cannot be empty. Preservation is sampled from zero to one, steering uses a logarithmic scale from 0.0001 to 1.0, overcorrection ranges from zero to 1.3, and the neighbor count ranges from one to at most fifteen or the size of the smaller example set.

The implementation also has to make repeated objective calls practical. Materializing a complete effective weight matrix inside every closure call would add expensive allocation and matrix work. The code first computes the base product between module input and the original transposed weight. It then adds the low-rank update through two smaller matrix products. When row-magnitude preservation is enabled, it scales the result using the original and modified row norms, then restores the bias. This keeps the optimizer loop centered on the adapter matrices rather than rebuilding the dense model matrix.

The inner optimizer is L-BFGS with learning rate 1.0, five outer steps, up to twenty inner iterations per step, a history of ten updates, and strong Wolfe line search. The A matrix is initialized from a dedicated deterministic generator so that the result does not depend on unrelated uses of the global random state. Each layer's gradients are cleared after optimization. A reset path returns the model to its initial adapter state before the next trial.

Directional ablation and MPOA

The second modifier follows the better-known abliteration route. It computes per-layer mean residuals for good and bad prompts and uses their difference as a behavioral direction. By default, that direction is orthogonalized against the good direction, so only the component unrelated to normal behavior is removed. The direction can be selected by interpolating between two global layer positions or computed independently for each layer.

Rather than applying a single constant edit everywhere, the modifier searches a weight kernel for each modifiable component. The kernel has a maximum weight, a maximum position, a minimum weight, and a distance over which the weight decays. Layers beyond that distance are left untouched. The ranges in the code come from prior experiments and are not presented as universal constants. The MLP lower bound is allowed to be negative before being clamped to zero, which gives the optimizer a real opportunity to disable MLP ablation. The source comments explain why: attention-output edits are often enough to weaken refusals, while modifying the MLP too aggressively can cause more capability damage.

Without row normalization, a rank-one adapter is sufficient to represent subtraction along one direction. Full row-magnitude preservation introduces a nonlinear normalization step that a rank-one decomposition cannot reproduce exactly. The project therefore uses a rank-three adapter by default when full normalization is active. It calls the combination Magnitude-Preserving Orthogonal Ablation, or MPOA. The name should not be read as perfect preservation. The source explicitly describes the low-rank reconstruction as approximate, and the final model still needs behavioral evaluation.

Two editing mechanisms and their design tradeoffs

Property	ARA	Directional ablation and MPOA
Primary signal	Module input and output neighborhoods for good and bad prompts	Per-layer residual-mean difference as a direction
Default rank	50	1, or 3 with full row-magnitude preservation
Search space	Layer range, preservation and steering weights, overcorrection, neighbors	Global or per-layer direction plus component-specific weight kernels
Computation	Runs L-BFGS over module outputs for every trial	Computes directions first, then projects and approximates the update
Interpretive model	Local movement in representation space	A more direct removal of one behavioral component

The configuration key is named modifiers, but the current runtime requires exactly one modifier in the list. ARA and directional ablation are alternatives, not layers that can be composed in a single processing run. This constraint keeps each trial's objective and reproduction record easier to reason about, while also preventing two independently tuned parameter spaces from fighting over the same weights.

Scorers define the target

Scorers are not passive evaluation tools in Heretic. Their values become the Optuna objectives, which means every blind spot becomes something the optimizer can exploit. The default configuration minimizes two scores. KeywordRate measures the fraction of harmful dataset prompts whose responses contain one of 33 refusal or disclaimer markers. KLDivergence measures how far the modified model's first-token probability distribution has moved from the baseline on a harmless dataset.

Keyword matching is deliberately simple. It lowercases the response, removes asterisks, normalizes curly apostrophes, compresses whitespace, and checks for substrings. Empty responses are counted as matches so that an optimizer cannot improve the refusal score by producing nothing. That anti-cheating rule is useful, but it does not solve the semantic problem. A response can refuse without using any marker. A model can also adopt a terse, evasive, or incomplete style that lowers the lexical rate while failing the user's request in a different way.

The optional BERT classifier scorer can recognize full or partial refusals using a text-classification model and a configurable label set. It reduces dependence on exact wording, but it replaces one proxy with another. The classifier has a training distribution, a threshold, a version, and label definitions that may not match the deployment domain. Heretic can pin a model commit and record the resolved classifier version, which helps reproducibility, but cannot turn a biased classifier into ground truth.

The KL scorer compares only the next-token distribution at the first generated position. It is a cheap regularizer against large representation drift. It is not a measure of whole-response preservation. Two models can have nearly identical first-token probabilities while diverging later in reasoning, code, formatting, language choice, or tool-call syntax. Conversely, a harmless first-token difference may have little practical consequence. Standard Language Model Evaluation Harness tasks can be configured as additional scorers or final checks, but they must be selected for the capability the deployment actually needs.

What the default evidence can and cannot establish

Signal	Measured quantity	Unsupported conclusion
Refusal keyword rate	Presence of configured strings; empty output counts as a match	Whether the model answered, remained safe, or merely changed wording
Refusal classifier rate	Classifier decision over a prompt and response pair	Coverage outside the classifier's labels, domains, and threshold
First-token KL	Baseline-to-modified divergence at one generation position	Preservation of multi-step reasoning, long context, or tool use
Configured benchmark	One task aggregate under chosen decoding settings	General capability, safety, or robustness outside that task

A useful experiment therefore adds outcomes that the optimizer is not directly gaming. Compare baseline and edited models on human review of representative requests, completion rate, empty-response rate, repetition, language consistency, long-form coherence, code execution, mathematical reasoning, tool-call validity, and refusal semantics. Use the same prompts and decoding settings for both models and report variance across runs. For a safety-sensitive deployment, treat model editing as one layer among many. External policy checks, permissions, logging, rate limits, and human escalation cannot be replaced by an objective that only counts refusal phrases.

Searching a frontier instead of one score

Heretic uses Optuna's multivariate Tree-structured Parzen Estimator sampler. The default run contains 100 trials and reserves 30 startup trials for broader exploration. Each trial's parameters and scores are written into a journal checkpoint, which allows a later run to continue the same study. A trial applies one modifier, evaluates all configured scorers, and can be pruned if it fails or cannot complete. The result is not a single scalar winner but a set of trials that cannot be improved on one objective without worsening another.

This Pareto-first approach is the right abstraction for behavior editing. One candidate may suppress the lexical refusal rate most strongly at the cost of a larger distribution shift. Another may preserve the baseline more closely while leaving some refusal behavior intact. A third may sit between them and generalize better on prompts outside the optimization set. The interface exposes these tradeoffs so that the user can select a candidate and immediately inspect it through chat or benchmarks. The search can be extended without discarding earlier trials.

The frontier is only as meaningful as its axes. If the only active scorer is lexical refusal rate, the search may find a model that changes wording without changing policy. If KL is the only preservation signal, the search may select a model that barely changes behavior. If a benchmark is optimized repeatedly on the same test set, the team can overfit to that task. The design does not eliminate this responsibility; it makes the objective explicit and auditable, which is more useful than hiding several unmeasured assumptions behind a single rank.

Reproduction and the Grimoire

Heretic separates a processed model from the evidence needed to recreate it. Export can produce only the low-rank adapter or merge that adapter into a full model. A reproducible upload can also include a reproduce directory containing human-readable instructions, the configuration, dependency information, the search checkpoint, weight hashes, and a machine-readable reproduce.json. The current manifest format is version 4 and records settings, modifier parameters, scores, datasets, package versions, optional system information, and hashes of exported weights.

heretic --reproduce path/to/reproduce.json skips the long search. It checks the environment against the manifest, reconstructs the settings and selected modifier, exports the model, and compares weight hashes. The claim of byte-for-byte reproduction is deliberately conditional: hardware, drivers, operating system, framework, package versions, and inputs must match. A different environment may produce a model that behaves similarly while failing the hash check. The manifest is evidence of a controlled reconstruction path, not proof that every stochastic or hardware-dependent operation is universally portable.

Reproducible publication has stricter eligibility than ordinary export. The base model must come from Hugging Face, datasets must resolve to fixed commits, and the modifier and scorers must be built-in plugins marked as reproducible. Local paths and external plugins are excluded because they can disappear or change after publication. These restrictions are practical. A manifest that points to a mutable local file is a recipe, not a reconstruction contract.

Because a manifest is only around nine kilobytes, many manifests can form a portable archive without carrying all model weights. heretic --collect-reproducibles scans public Heretic models and downloads their manifests into a local collection. The Heretic Grimoire is the public, distributed view of that archive. Collection is append-only: rerunning the command updates the local catalog but does not delete an older manifest when its model disappears from Hugging Face. That preserves a historical record of method versions and environments, while still depending on the original base model and datasets remaining accessible.

Architecture support and operational limits

The repository supports most dense causal language models and also contains module discovery for several multimodal language backbones, mixture-of-experts layouts, and hybrid architectures. It does not assume that every model has the same attention or MLP names. Discovery looks for standard attention output projections, linear-attention output projections, vendor-specific MLP down projections, shared experts, and routed experts. This is wider than a one-model script, but it is not a universal graph editor. A new architecture can produce missing modules, an incorrect component selection, or an adapter that cannot be restored exactly.

Memory is the first hard limit. On-the-fly 4-bit quantization through bitsandbytes lowers the weights' storage cost at a small accuracy cost. Merging a quantized model is a different operation: Heretic must reload the base and dequantize the full model, which can require roughly three times the parameter count in gigabytes of system RAM. Even without full training, captured outputs, gradients, optimizer state, and generation buffers accumulate. Moving analysis tensors to CPU earlier reduces VRAM pressure but increases transfer time.

Automatic batch-size probing measures generation speed on one short workload. It does not reserve enough memory for every modifier, context length, or dataset batch that follows. Thinking models need extra wrapper handling or the scorer may inspect a truncated or prefix-polluted response. Vision-language models may be edited only in the language backbone, leaving questions about image encoder participation, processor files, and chat-template export. Mixture-of-experts models add routing complexity: a prompt activates only some experts, so captured outputs and effective updates are not equivalent to editing every expert uniformly.

The configuration layer is another migration boundary. The current version uses plugin namespaces and named instances, so multiple scorers of the same class can use different datasets and settings. Old fields may be rejected or assigned a different interpretation. A user should run a help command or a small validation model before committing hours to a large search. The project does not restart or modify a deployed service as part of an edit; the generated artifacts are ordinary model files that must pass a separate deployment process.

A practical evaluation protocol

A defensible test begins with a held-out prompt set that was not used by the scorer or the search. It includes ordinary requests, edge cases near the intended behavior boundary, multilingual prompts, long prompts, tool-shaped outputs, code tasks, and requests that should still be declined. The baseline and modified model receive exactly the same prompts and decoding parameters. A reviewer records whether the request was completed, refused, partially answered, changed topic, hallucinated, or violated policy. Automated metrics are reported alongside those labels rather than substituted for them.

The next layer checks unintended drift. Run a small benchmark suite that matches the deployment: reasoning, code, structured output, and long-context tasks if those are relevant. Compare not only averages but failure tails and variance. A model whose average score is stable can still become much worse on a narrow but operationally important slice. Examine language switching, repeated opening phrases, empty responses, excessive verbosity, and degradation after quantization. Repeat the evaluation with the exported adapter and the merged model, because export strategy can change memory behavior and numerical results.

Safety review should be separate from capability review. The default configuration is explicitly designed to weaken refusal boundaries. Even when the target is a legitimate local research setup, the resulting model may expose information or instructions that the original model withheld. The correct response is not to relabel that change as “more helpful” or “safer.” It is to document the changed boundary, restrict tool access, retain external moderation where appropriate, and make the model's limitations visible to users. High-stakes domains such as medicine, law, finance, or autonomous control require independent review rather than a single refusal-rate improvement.

What Heretic contributes

The strongest part of Heretic is not a magic layer index or a universal direction vector. It is the decomposition of behavior editing into four auditable pieces. Contrasting data defines the target behavior. A modifier defines the family of weight changes that can express the edit. Scorers define the surrogate objective that search will optimize. A reproduction manifest defines the environment and inputs needed to rebuild the result. If any piece is vague, the final model will amplify that vagueness.

ARA contributes a relatively flexible neighborhood objective and a reusable low-rank export path. Directional ablation and MPOA contribute a lower-rank, more interpretable baseline. Optuna contributes multi-objective search rather than a single hidden score. The plugin system makes it possible to change the target without editing the core package. The Grimoire treats model edits as records that can be compared and reconstructed, not as opaque binary releases.

The limits are equally concrete. Keyword refusal scoring is lexical. First-token KL is not a capability benchmark. The default search can find a shortcut that satisfies its objectives without satisfying the user's intent. Architecture discovery is broad but finite. Quantization and export can change behavior. Reproducibility depends on an exact environment and immutable inputs. Community upload counts and third-party leaderboard comparisons are not substitutes for a controlled evaluation of the model a team plans to deploy.

Conclusion

Heretic is best understood as a compact experiment framework for changing a model's behavioral priors while leaving the base model in place. It turns refusal suppression or style adjustment into a measurable search problem, exposes the tradeoff frontier, and records enough information to reconstruct a selected edit. That is a meaningful engineering contribution, especially for local model researchers who need to compare layer ranges, strengths, and preservation strategies instead of guessing at a direction vector.

It is not a shortcut to a universally better model. The right questions are empirical: Which behavior boundary moved, what evidence shows that it moved, what capabilities changed outside the scorer, can the artifact be rebuilt, and does the result remain acceptable under the deployment's policy? Heretic provides a strong scaffold for asking those questions. A lower refusal rate, a persuasive chat transcript, or a large collection of uploaded derivatives is not enough on its own.