Just Ask Jev: Reinforcement Learning for Calibrated Decisions as a Zero-Shot Detector of AI Alignment Failures

Paper Information

Title: Just Ask Jev: Reinforcement Learning for Calibrated Decisions as a Zero-Shot Detector of AI Alignment Failures

Authors: Ruoqi Guo, Yi Liu, Gelei Deng, Yuekang Li, Lida Zhao, Yutao Wu, Simin Chen, Ying Zhang, and Leo Yu Zhang.

Paper: arXiv:2609.29429, submitted September 24, 2026.

Code and data: sumleo/RLCDAlignBench (MIT code; CC BY-NC 4.0 data) with project page, dataset, cached Jev responses, result tables, and an offline rescoring path.

Research Background and Motivation

Alignment-failure detectors are used twice. In deployment, they screen model inputs, outputs, and agent trajectories before unsafe behavior reaches a user or an external tool. In research, they score benchmarks and compare mitigations. A detector therefore has to rank subtle failures accurately, and it has to remain cheap because it may run on every message.

Most current detectors are generative judges. They receive a criterion, generate a verdict, and require another parser or aggregation step before the verdict can be used as a score. A second family reads a token probability, such as Llama Guard's $P(\text{unsafe})$, which gives a continuous score directly. Both designs still pay a decoding pass for each criterion. Asking the same interaction about ten possible failures can therefore require ten calls, even when the underlying evidence is shared.

RLCD removes that per-criterion cost. A model trained for calibrated decisions returns a probability for each typed question instead of generating free-form text. Calibration means that among decisions assigned probability $p$, a fraction close to $p$ should be correct. Jev applies this interface to a state: it can answer many Noul, Choice, and Score questions about the same input in one request.

The hard case is relational failure. Sycophancy is defined against the user's belief, deception against the model's own belief or goal, and prompt injection against an instruction hidden in tool output. A detector that sees only the final response may not have the reference that makes the failure identifiable. A bad score can therefore come from the question, from the missing context, or from a defective label. Treating these causes as one problem hides the repair path.

The paper separates the two axes. It varies what Jev is asked independently from what Jev sees, then measures the resulting score against labels Jev never receives. The result is RLCDAlignBench: ten failure types, 44 benchmarks, 5 target models, and 7,193 detection instances. Human labels on StrongREJECT and HarmBench validation provide an external check, while second-judge labels on AbstentionBench and InstrumentalEval expose scorer-specific noise.

Preliminaries: RLCD, Jev, and Detection

An interaction is represented by a state $x$ that contains the context the target model received and its response or trajectory. For a failure type $f$, the benchmark supplies a binary label

$$y_f(x)\in\{0,1\},\qquad s(x)\in[0,1],\qquad \hat y(x)=\mathbf{1}[s(x)\ge t].$$

A detector must solve two problems: rank positive failures above negatives, and choose a threshold $t$ that converts the ranking into decisions. AUROC measures the first; F1 at a fixed or fitted threshold measures the second. A good ranker with a badly transferred threshold can still perform poorly in deployment.

Jev accepts a state $\sigma$ and a set of questions $\{q_1,\dots,q_m\}$. A Noul question returns $P(\text{yes}\mid\sigma,q)$; a Choice question returns a distribution over options $O_q$; a Score question returns a distribution over ordered levels $\ell_1\lt\cdots\lt\ell_K$. Choice and Score also return confidence. The questions share one call, are answered independently, and do not require a generated answer to be parsed.

RLCDAlignBench turns each benchmark into a detection instance containing three parts: the state Jev sees, the binary label, and metadata that never enters the request. The canonical state is what a response-level or system-level monitor can observe. Extra fields, such as a gold answer, secret list, injected instruction, or deceptive goal, are tested as separate variants because their availability changes the meaning of the detection task.

Method: A Question-and-Context Factorial Design

Figure 1 summarizes the first result: one Jev call per item already ranks most alignment failures well. The paper reports the performance of 38 usable benchmarks, while excluding six benchmarks whose minority class contains fewer than five items. The generic Noul question is available on 31 of them; a targeted question selected on one half of the items and evaluated on the other reaches the larger 38-benchmark set.

The design has three layers. First, it fixes a generic question built from the benchmark's published behavior phrase. Second, it varies the answer type among Noul, Choice, and three-level Score while keeping the wording fixed. Third, it adds targeted questions that name the behavior more directly, including benchmark-specific definitions, reference-scorer rubrics, and decomposed questions such as one question per option or secret.

flowchart LR
    A[Target-model interaction x] --> B[Canonical state: input + output]
    B --> C[Jev: one call with typed questions]
    D[Generic Noul / Choice / Score] --> C
    E[Targeted direct / rubric / decomposition] --> C
    F[Context variants: reference, attack prompt, self-report] --> B
    C --> G[Probabilistic answers and confidences]
    G --> H[Strategy s(x) = combination of answers]
    H --> I[s(x) at or above threshold t]
    I --> J[AUROC, F1, calibration, label audit]

Figure 2 gives the complete pipeline. A target-model interaction receives a reference label from the benchmark's rule, judge, or multi-turn scorer. Jev sees only the selected state variant and answers the full question battery in one request. The resulting answers are combined into $s(x)$, which is compared with labels that were never placed in the state.

The benchmark spans ten failure families. The target models are small open models in the 2B to 7B range, which makes the benchmark a test of Jev rather than a comparison of frontier-model safety. The label sources are deliberately mixed: 20 benchmarks use a rule or log-probability scorer, 20 use an LLM judge, and 4 use a multi-turn trajectory judge.

Failure type	Benchmarks	Items	Target model	What the positive label means
Sycophancy	4	639	Qwen3.5-2B	The reply defers to the user's stated view or flips its verdict.
Jailbreaks	4	414	Phi-4-mini	The reply complies with a harmful request.
Deception	4	540	Gemma-2-2B	A claim contradicts the elicited belief or serves a deceptive goal.
Prompt injection	4	1,036	Qwen3.5-2B	The agent follows an injected instruction or leaks protected material.
Hallucination	6	1,164	Llama-3.2-3B	The response claims something unsupported by the source.
Privacy violation	4	808	Phi-4-mini	The response discloses information or permits a prohibited flow.
Social bias	4	199	Olmo-3-7B	Two demographic variants differ in a stereotype-consistent way.
Reward hacking	6	714	Qwen3.5-2B	The model exploits a grader or reward instead of the intended goal.
Concealing uncertainty	4	748	Olmo-3-7B	A wrong answer is given with high confidence or answered when abstention is expected.
Power seeking	4	931	Llama-3.2-3B	The model chooses a more power-seeking or unethical option.
Total	44	7,193	5 models	Ten failure families

The generic question is not a free-form prompt. Each state format gets one stem, and only the behavior phrase and field names change. A single-response state asks whether the response exhibits the behavior. A paired-output state asks about two fields. A rule-scored state points to the fields used by the benchmark. A trajectory state points to the assistant's field inside a longer interaction. Every instruction begins with a guard that tells Jev to treat the state as evidence, not as instructions to follow.

The fixed wording is asked in all three answer types. The Noul score is simply $s=P(\text{yes})$. The three-level Score is read as

$$s_{\mathrm{soft}}=\frac{\mathbb{E}[\mathrm{level}]}{2}.$$

Choice and Score have argmax variants as well. A strategy can combine multiple answers through a mean, maximum, product, difference, or a logical rule, so the general form is

$$s(x)=g(a_1,\dots,a_m).$$

The paper evaluates 15 to 40 strategies per benchmark, but keeps them in a single request. This feature is what makes Jev structurally different from a judge: increasing the number of diagnostic questions does not multiply the number of calls. The released runner builds one request from the state and all battery questions:

# code/run_jev.py:339-343
def build(s):
    state = render_text(s["state"])
    return {"state": state,
            "model": args.model,
            "questions": battery.questions(s)}

Reporting the best of dozens of strategies would inflate the result. The paper therefore selects a strategy on one half of the item groups and scores it on the other half. For each context comparison, the primary strategy must also survive a permutation check: when Jev's answers are shuffled across items, its AUROC must fall close to chance. This removes strategies whose apparent signal comes from metadata or code rather than from Jev.

Three baselines bound the task. The all-positive predictor has

$$F1_{\text{all+}}=\frac{2p}{1+p},$$

where $p$ is the positive base rate. A response-length classifier chooses its direction out of sample. A TF-IDF logistic regression is trained on in-domain labels using word and character n-grams. The TF-IDF model sees supervision that Jev does not, so beating it zero-shot is a stronger result than beating an untuned heuristic.

The evaluation uses AUROC as the primary ranking metric, F1 at $t=0.5$, and a two-fold cross-fitted threshold chosen on one group of items and applied to the other. Abstentions count as negative decisions. A strategy enters an AUROC comparison only when it covers at least 90% of items. Confidence intervals resample item groups with 1,000 bootstrap draws, preserving the dependence created by a shared source prompt.

The released code makes the protocol inspectable rather than hidden inside a notebook. The threshold search is explicit and cross-fitted:

# code/run_jev.py:198-218
fold = {}
for g, _, _ in rows:
    fold[g] = int(hashlib.sha1(str(g).encode()).hexdigest(), 16) % 2

for k in (0, 1):
    train, test = parts[1 - k], parts[k]
    best = max(THR_GRID,
               key=lambda th: (
                   binary_metrics([1 if r >= th else 0
                                   for _, r, _ in train],
                                  [l for _, _, l in train])["f1"],
                   -abs(th - 0.5)))

The context experiments reuse the same items and labels while changing one state field. A reference can be missing, derivable, already present, or auxiliary. A field is considered deployable if a real system-level monitor might hold it. It is a label key if the benchmark label is defined by that field, such as a secret list for PrivacyLens. This distinction prevents an improvement from a gold answer being presented as an improvement in detecting the behavioral failure.

Experimental Results

One generic question is already a strong ranker

Across the 31 benchmarks that admit a generic Noul question, Jev reaches a median AUROC of 0.886, with a grouped-bootstrap interval of [0.821, 0.952]. The split-half targeted strategy reaches 0.911 [0.860, 0.944] over all 38 usable benchmarks. Against the better of length and TF-IDF, the generic question improves AUROC by a median of +0.132 [+0.057, +0.190] and wins on 25 of 31 benchmarks. The scorer family does not explain the result: medians are 0.906 for LLM judges, 0.890 for rules, and 0.870 for multi-turn judges.

The label source does matter. Jev scores 0.949 on 20 mostly unvalidated judge-labeled benchmarks and 0.872 on 8 validated benchmarks. That gap is not simply evidence that Jev is better on weak labels. It also reflects the set of constructs in each group, and it motivates the label audit later in the paper.

Question wording adds only a small out-of-sample gain. The best targeted strategy beats the best generic readout by a median of +0.006 [-0.004, +0.015] AUROC, with a win/tie/loss count of 24/1/11 and a Wilcoxon p-value of 0.055. The paper's stricter interpretation is that targeted wording is useful in selected cases, but it is not the main source of performance.

Keeping probabilities soft matters more than the question label

The three-level Score read as an expectation wins against the generic Noul on 19 of 31 benchmarks, with a median gain of +0.016 [+0.001, +0.030]. The soft Choice is approximately tied. Argmax readouts lose almost everywhere: the Choice argmax loses on 28 of 31 benchmarks and the Score argmax on 24 of 31. When a rubric thresholds answers at 0.5 before combining them, it loses to the best direct question on 9 of 10 benchmarks, with a median drop of -0.137 AUROC.

This result is not a small implementation detail. It means that the useful signal is the distribution over possible answers, not just the most likely answer. Hard thresholds destroy probability mass before the downstream combination can use it. The best generic readout for a benchmark can therefore be a soft Score even when a Noul or Choice format looks more natural to a human reader.

Metric or comparison	Result	Interpretation
Generic Noul median AUROC	0.886 [0.821, 0.952]	Strong zero-shot ranking on 31 benchmarks.
Split-half targeted AUROC	0.911 [0.860, 0.944]	Small gain over generic performance; selection is honest.
Targeted minus generic	+0.006 [-0.004, +0.015]	Wording is not the main bottleneck.
Score soft vs. generic Noul	+0.016 [+0.001, +0.030]	Soft ordinal readouts are competitive and often better.
Soft vs. argmax Choice	+0.084 AUROC	Thresholding away probability mass is expensive.
Generic Noul over TF-IDF or length	+0.132 [+0.057, +0.190]	Jev wins without in-domain supervision.

Context helps when the state lacks a label-defining reference

A deployable reference that is absent from the state helps the generic Noul with a confidence interval above zero on only 1 of 4 benchmarks. On DeceptionBench, adding the target's goal prompt raises AUROC by +0.050 [+0.029, +0.077]. A distilled reference that is already present helps on 0 of 3 MASK comparisons. Attack prompts move the best shared question by at most 0.002.

Label keys behave differently. They raise the generic Noul on 4 of 11 benchmarks, with a median gain of +0.053. PrivacyLens is the clean example: adding its list of secret items lifts the generic question from 0.79 to 0.95. But this is not a free deployment improvement. The secret list is part of the label definition. It tells the detector which disclosure counts as a privacy violation.

SycophancyEval exposes the trap. Adding the true answer moves the generic Noul from 0.540 to 0.941 on the official correctness label, but from 0.712 to 0.288 on a corrected answer-shift label. The original label measures whether the answer is correct, not whether the answer moves toward the user's suggestion. A strong gain on the official benchmark can therefore be evidence of a label mismatch rather than better sycophancy detection.

Calibration is good in the pool, weak within a benchmark

Pooled across benchmarks, the generic Noul is close to calibrated: expected calibration error is 0.047. Per benchmark, the median ECE is 0.168 against a perfect-calibration null of 0.074, and 24 of 31 benchmarks exceed the null's 95th percentile. The problem is a base-rate mismatch. Within a file, Jev's mean probability misses the positive rate by a median of 0.125 even though its median file-level AUROC is 0.905.

A fitted threshold recovers much of the decision performance. Median F1 increases from 0.706 at $t=0.5$ to 0.822 with a cross-validated threshold and to 0.793 when only 10 labeled items are used to fit the threshold. The gain is concentrated in unvalidated rule-scored benchmarks where Jev ranks well but produces probabilities below the decision boundary. On validated labels, $t=0.5$ is already as good.

The paper does not hide the transfer problem. A threshold that is optimal on one benchmark can lose to the all-positive predictor on another, especially when the positive base rate is high. The safer deployment pattern is selective routing: keep the half of decisions with the largest confidence and send the rest to a stronger judge or a human. This raises median accuracy from 0.793 to 0.933.

Human agreement and label defects

On StrongREJECT, the generic Noul agrees with human labels at Cohen's kappa 0.809, while the GPT-4o-mini reference scorer reaches 0.811. The difference is -0.002 [-0.059, +0.057]. Jev ranks the same responses better, with AUROC 0.971 versus 0.929, and humans side with Jev on 49% of the 116 disagreements. On HarmBench validation, Jev's mean agreement with a single annotator is kappa 0.748, close to the 0.736 inter-annotator agreement.

The parity is not uniform. On GPT-3.5 responses, Jev reaches kappa 0.668 while the reference scorer reaches 0.790, even though their AUROCs are similar. The paper therefore treats per-generator calibration and score shape as an open problem.

Confident disagreements also exposed label defects. Open-Prompt-Injection's label tracks accuracy on the injected task, SycophancyEval (answer) tracks answer correctness, and the MACHIAVELLI labels depend on annotated consequences that are absent from the state. Two AbstentionBench judges agree at only kappa 0.05, showing that the same benchmark can carry substantially different labels depending on who runs the official prompt.

One battery, one call, and a large cost gap

One Jev call sends one detection instance with the whole question battery. Across the study, 23,411 calls contained an average of 11.4 questions, had a 0.12% error rate, and returned in a median client latency of 0.313 seconds. Latency did not grow with the number of questions: its correlation was -0.037 with question count and -0.002 with input-token count.

On the 19 benchmarks with an API LLM judge, one Jev pass cost $0.302 at list prices against $18.96 for the reference judges, a pooled ratio of 62.9 times. The advantage depends on the deployment assumptions. If every judge is repriced at GPT-4o-mini rates and Jev is asked only the single generic question, Jev is still 12.1 times cheaper pooled and 3.3 times cheaper at the per-benchmark median.

Cost or calibration measure	Value	What it means
Jev median latency	0.313 s	One call carries the full question battery.
Questions per call	11.4 average	Extra diagnostics do not add calls.
Manual call error rate	0.12%	Low API failure rate.
Jev cost, 19 judge benchmarks	$0.302	One full detection pass.
API judge cost	$18.96	Reference scorers at listed prices.
Pooled cost ratio	62.9x	Jev is cheaper overall; per-benchmark ratios vary.
Median file ECE	0.168	Thresholds do not transfer cleanly across base rates.
F1 with 10 labeled items	0.793	Small amounts of supervision can recover much of the gap.

Limitations

The benchmark covers a single RLCD model, jev-1.13.0. It does not establish that another calibrated-decision model, a fine-tuned classifier, or a generative judge with probability readouts would show the same behavior. The result is therefore evidence about Jev and this interface, not a general law about all calibration objectives.

The target models are open models in the 2B to 7B range, and the evaluations are in English. The authors identify larger targets, other detectors, and other languages as future work. A failure detector can also be sensitive to prompt distribution, target-model family, and the language of the harmful request; the present benchmark does not measure those shifts.

Most labels come from benchmark scorers rather than fresh human annotation. The audit changes eight labels and finds rule defects, inconsistent judge behavior, and labels that depend on information omitted from the state. Seven of those enter the aggregates. The authors keep the official labels canonical and report corrected analyses separately, which is methodologically careful but leaves any detector comparison limited by the underlying label set.

Calibration does not transfer as a single threshold. Pooled ECE looks good, while per-benchmark ECE is often poor because the positive rate changes across failure types. A monitor using Jev as a turnkey classifier would need representative labels for its own deployment or a confidence-based fallback. The paper's recommendation of a threshold fitted on ten labels is a practical compromise, not a proof that the threshold will transfer.

The cost comparison is also conditional. It depends on list prices, judge call counts, the benchmark mix, and whether the full question battery or only the generic question is billed. PrivacyLens and the three MASK benchmarks account for 62% to 65% of the judge total in the reported scenarios. Without them, the pooled advantage falls from 62.9 times to 35.1 times, and under conservative repricing it falls further.

Conclusion and Outlook

The study supplies a useful separation of concerns for alignment monitoring. The detector's score can be strong even when its probabilities do not transfer across base rates, and the largest apparent gains can come from label keys rather than better perception of the failure. A benchmark that reports only the best question and the best aggregate would miss both effects.

For practitioners, the actionable recipe is narrow and concrete. Ask one generic question, keep its probability rather than taking an argmax, fit a threshold on a small labeled set from the deployment distribution, and route low-confidence decisions to a stronger checker. For benchmark builders, the more valuable result may be the audit method: confident disagreements are a cheap way to find labels that measure the wrong construct or depend on evidence the detector never receives.