An offline approach to fNIRS-guided reinforcement learning for robot behavior

Figure 1 — NEURO-LOOP overview. Offline fNIRS data trains a performance model that injects neural classification into the RL loop, augmenting (not replacing) priority, reward, or Q-values to guide policy learning.

1. Background & Motivation

Human-in-the-Loop RL (HITL-RL) trains/fine-tunes/aligns robot behavior via human feedback. Existing methods (learning from demonstration, preference learning, intervention learning) require explicit feedback via speech, physical demonstration, or interface preferences. Two problems arise:

• Cognitive overload: sustained attention and expert feedback burden the human teacher.
• Bias & exclusion: assumes implicit reactions follow training-set distributions, requiring users to move and speak — excluding people with motor/speech impairments.

Implicit HITL-RL uses passive feedback (gesture, expression, voice inflection) but still requires movement. This work uses brain signals as feedback, requiring no movement or speech. It uses fNIRS — slower hemodynamic signal but higher spatial resolution than EEG, more robust to certain artifacts.

Core question: Can offline fNIRS data (not real-time BCI) augment RL training, enabling deployment where real-time pipelines are impractical or data is limited?

2. Core Method

NEURO-LOOP injects fNIRS signals into DDPG-HER + Prioritized Experience Replay. Key design: augment not replace — neural signal modulates existing RL parameters rather than replacing the reward function.

2.1 Interaction Tasks

• Passive: participant observes robot performing fetch-and-place. ~50/50 optimal/suboptimal.
• Active: participant teleoperates robot. Suboptimal, noisier data (non-expert demos).
• Pooled: combines both.

2.2 Performance Model & Output Calibration

Three granularity levels mapping fNIRS features to agent optimality labels:

• Binary: Optimal/Not Optimal, $g_b \in \{0,1\}$
• Ternary: Optimal/Suboptimal/Not Optimal, $g_d \in \{0,1,2\}$
• Continuous: regressor predicting error vs near-optimal policy, $g_e \in \mathbb{R}$

Regressor output calibrated as:

$$n_t = (x_t + c_t \cdot \mu_X[0]) \cdot \beta$$

Classifier output calibrated as:

$$n_t = (x_t + c_t \cdot \mu_X[\hat{y}_t]) \cdot \beta$$

where $\hat{y}_t$ is raw model output, $x_t$ the parameter to augment (Reward/Priority/Q-value), $\mu_X$ describes expected parameter value per optimality class, $n_t$ the final value injected into RL.

2.3 Augmentation Techniques

2.4 Noise Injection

With probability $\tau$: flip classifier outputs or shift regressor outputs (random 0-1), testing robustness.

flowchart TB
    A[Offline fNIRS Data
21 participants] --> B[Performance Model
Binary/Ternary/Continuous]
    B --> C[Calibrate Output
n_t = f(x_t, c_t, mu_X)]
    C --> D{Augmentation Method}
    D --> E1[Prioritization
modulate trajectory priority]
    D --> E2[Reward Aug
modulate domain reward]
    D --> E3[Q-Aug
modulate state-action Q-value]
    D --> E4[All
combine three]
    E1 & E2 & E3 & E4 --> F[DDPG-HER + Prioritized Replay]
    F --> G[Robot Policy
Fetch-and-Place Task]
    style E3 fill:#dcfce7,stroke:#16a34a
    style E2 fill:#fee2e2,stroke:#dc2626
    style C fill:#e0e7ff,stroke:#2563eb

3. Experimental Results

Public fNIRS + robot behavior dataset, 21 participants (14F 11M, ages 19-27), evaluated every 10,000 steps, 25 episodes/evaluation. Metrics: total episode return and success rate.

Figure 2 — Binary performance model augmentation method comparison under best settings. Q-Aug and All consistently outperform Baseline. Prioritization ranks third; Reward Aug similar to Baseline. Error bars: 95% CI.

3.1 Augmentation Methods

• Q-Aug and All best: consistently outperform Baseline and all others — Q-value most directly influences policy.
• Prioritization third: modest but consistent improvement — competing with already-established TD-error ranking.
• Reward Aug ineffective: no significant improvement — consistent with [19]: neural feedback on reward pipeline performs like dense reward function.

3.2 Model Granularity

3.3 Noise Robustness

Figure 3 — Model noise effect. Conditions significantly above Baseline (Q-Aug) degrade with noise; conditions at Baseline level (Reward Aug) unaffected. Suggests improvements stem directly from neural signal.

• Q-Aug: degrades as noise increases.
• Reward Aug: virtually unaffected.
• Setting noise to 0.5 makes binary converge to ternary — confirming ternary underperformance due to noisier output.

4. Main Contributions

• NEURO-LOOP framework: First systematic framework for injecting offline fNIRS brain signals into RL training, testing augmentation methods, model granularity, noise robustness, and finetune checkpoints.
• Augment not replace: Neural signal modulates existing RL parameters (priority/reward/Q-value) — Q-value best, reward ineffective.
• Offline viability: Works purely offline, no real-time BCI hardware/calibration overhead — deployable where real-time pipelines impractical.
• Noise robustness analysis: Systematically characterizes tolerable noise levels — gains erode as noise increases.

5.

门控模块输出预测

$$ \mathcal{M}_{g}(f_{t})=\hat{y}_{t} $$

置信度估计

$$ c_{t}=1-\hat{e}_{t} $$

噪声调整公式

$$ n_{t}=(x_{t}+c_{t}\cdot\mu_{X}[\hat{y_{t}}])\cdot\beta $$

Limitations & Future Work

Author-stated limitations:

• Agent performance labels are a function of agent optimality, not guaranteed to reflect participant's true judgment — assumes alignment with near-optimal policy.
• 4-second neural feature windows sidestep the credit assignment problem; temporal delay between fNIRS hemodynamic response and agent interaction is not standardized.
• No comparison against other HITL approaches — unclear what neural signal contributes beyond existing feedback channels.

Analysis:

The key value is proving offline brain signals can augment RL training — Q-value augmentation's consistent improvement and noise-degradation pattern convincingly show effects stem from neural signal, not framework artifacts. However, offline methods rely on pre-recorded data mapped to specific tasks; transferring to new tasks requires re-collecting brain data. The 4-second window is an engineering compromise; fNIRS hemodynamic delay (6-8s peak) makes fine causal attribution difficult. The most critical future direction is extending to online real-time settings and physical robots — the leap from simulation to real deployment.

6. Conclusion

This work shows fNIRS signals can update a robot learning algorithm and improve training efficiency. NEURO-LOOP evaluates different augmentation methods, model granularity, and noise robustness. Results show the pipeline works purely offline — a practical alternative for settings where real-time BCI is impractical. Q-value augmentation works best, reward augmentation has no effect, and augment-not-replace is the core design principle.

The framework provides a starting point for extending to real-time, physically embodied human-robot interaction. Key challenges: aligning performance labels with user judgment, standardizing fNIRS delay handling, systematic comparison with other HITL paradigms.