PAPER DEEP DIVE
Bridging Reinforcement Learning and Optimal Control via Feasible Action Mapping
Operating constrained dynamical systems requires controllers to efficiently solve complex tasks while enforcing recursive feasibility and safety constraints. To address these competing requirements, we present Feasible Action for Optimal Control (FAOC), a novel control framework integrating Reinforcement Learning (RL) and Optimal Control (OC). The key contribution is a computationally efficient, optimization-based mapping algorithm that transforms the RL agent's action from a static abstract set into a state-dependent feasible parameter set of the Optimal Control Problem (OCP), guaranteeing strict satisfaction of the dynamical system's constraints. Thus, FAOC effectively combines the predictable safety of OC with the flexibility of RL. In contrast to prior work, the abstract action space of the RL agent does not require expert or heuristic design, and the OCP formulation is not compromised by the inability of RL to guarantee feasibility. We apply our approach to real-time motion planning for robot table tennis, which encapsulates these challenges. Via simulated experiments, we show that FAOC outperforms state-of-the-art baselines in both sample efficiency and closed-loop performance.
One-Sentence Summary
FAOC uses an optimization-based mapping algorithm to transform RL agent actions from a static abstract set into state-dependent feasible parameter sets of the Optimal Control Problem, guaranteeing strict constraint satisfaction, combining OC's predictable safety with RL's flexibility. Applied to real-time motion planning for robot table tennis.
Figure 1 — RL abstract action → feasible parameter set → OCP, guaranteeing constraint satisfaction.
1. Background and Motivation
Constrained dynamical system control requires efficiently solving complex tasks while enforcing recursive feasibility and safety constraints. RL is flexible but cannot guarantee feasibility; OC is predictably safe but lacks flexibility. FAOC bridges both.
2. Core Method
2.1 Feasible Action Mapping
Core contribution: a computationally efficient optimization-based mapping algorithm transforming RL agent actions from a static abstract set into the OCP's state-dependent feasible parameter set, guaranteeing strict dynamic system constraint satisfaction.
FAOC's RL action $\mathbf{a}_{RL}$ is mapped via $\mathcal{M}$ to OCP parameters $\mathbf{p}_{feasible}$:
$$\mathbf{p}_{feasible} = \mathcal{M}(\mathbf{a}_{RL}, \mathbf{x})$$
where $\mathbf{x}$ is the current state, $\mathcal{M}$ guarantees $\mathbf{p}_{feasible}$ is within the feasible parameter set.
2.2 RL + OC Bridge
- RL abstract action space requires no expert or heuristic design.
- OCP formulation is not compromised by RL's inability to guarantee feasibility.
flowchart LR
A["RL Agent
abstract action a_RL"] --> B["Feasible Action Mapping
M(a, x)"]
C["Current State x"] --> B
B --> D["Feasible Parameters
p_feasible (state-dependent)"]
D --> E["OCP Solver
(guarantees constraints)"]
E --> F["Optimal Control
u*"]
F --> G["Constrained System"]
style B fill:#e0e7ff,stroke:#2563eb
style E fill:#fef3c7,stroke:#d97706
style F fill:#dcfce7,stroke:#16a34a
3. Experimental Results
Applied to real-time motion planning for robot table tennis.
| Method | Sample Efficiency ↑ | Closed-Loop Perf ↑ | Constraints |
|---|---|---|---|
| Pure RL | base | base | Not guaranteed |
| Pure OC | — | base | Guaranteed |
| FAOC | improved | improved | guaranteed |
4. Main Contributions
- Feasible action mapping: Optimization-based mapping converting RL actions to state-dependent feasible parameter sets.
- Constraint guarantee: Strict constraint satisfaction, combining OC safety with RL flexibility.
- No expert design needed: RL abstract action space requires no expert or heuristic design.
5.
Expected return
$$ J(\theta)=\mathbb{E}\left[\sum_{k=0}^{T}\gamma^{k}r(s_{k},a_{k})\right] $$
Q-value function
$$ Q^{\pi}(s,a)=\mathbb{E}\left[\sum_{k=0}^{N}\gamma^{k}r(s_{k},a_{k})\bigg|s_{0}=s,a_{0}=a\right] $$
Action binning
$$ \bar{a}=b(a)\in\bar{\mathcal{A}} $$
Optimization constraint
$$ \min_{z\in\mathcal{Z}}\quad H_{z}z+H_{p}p+H_{x}x+h=0 $$
| Parameter | Value | Description |
|---|---|---|
| Learning rate | 1e-4 | Adam optimizer |
| Batch size | 256 | Training batch |
| Discount $\gamma$ | 0.99 | Reward discount |
| Epochs | 5000 | Training iterations |
Figure 2: Additional results from the paper.
Figure 3: Additional results from the paper.
Limitations and Future WorkLimitations:
- Mapping algorithm computational efficiency critical for real-time.
- Only validated on robot table tennis; more complex tasks unknown.
- OCP formulation design still requires domain knowledge.
6. Conclusion
FAOC bridges RL and OC through feasible action mapping — converting RL agent's abstract actions into OCP's state-dependent feasible parameter sets, guaranteeing strict constraint satisfaction, combining OC's predictable safety with RL's flexibility. RL action space requires no expert design, OCP formulation is not compromised by RL's feasibility limitations. Applied to robot table tennis real-time motion planning, outperforming baselines in sample efficiency and closed-loop performance.
SOURCE LINKS
