Synthesis and Deployment of
Maximal Control Barrier Functions
through Adversarial Reinforcement Learning
We introduce a framework for synthesizing and deploying robust control barrier functions (CBFs) for high-dimensional black-box systems under bounded uncertainty. We show that the solution to the safety Isaacs equation is a valid discrete-time CBF (DCBF) and then approximate this value function using adversarial reinforcement learning (RL).
The resulting safety filter can be synthesized and deployed at scale through black-box interactions with the system, without explicit knowledge of its dynamics.
Contributions
We formally show that the safety value function—solution to the dynamic programming safety Isaacs equation—is a valid robust DCBF on the maximal robust safe set, and that its state–action lift yields a novel CBF constraint for smooth safety filtering under uncertainty.
Leveraging reachability-based adversarial RL, we develop a scalable robust Q-CBF synthesis and deployment pipeline for high-dimensional systems without requiring explicit dynamics, control-affine assumptions, prescribed uncertainty structure, or manual barrier function design.
We validate our robust Q-CBF on a disturbed inverted pendulum and a high-fidelity quadrupedal locomotion simulation governed by 36-dimensional black-box dynamics. On the pendulum, the learned robust Q-CBF nearly recovers the maximal robust safe set and is substantially less conservative than barrier-based baselines. On the quadruped, it reliably enforces safety under adversarial uncertainty realizations while achieving effective forward locomotion.
Exact-Value Q-CBF Theory
Scalable Synthesis and Deployment on Black-Box Systems
Scalable value approximation via game-theoretic RL
We synthesize neural safety values and policies through adversarial reinforcement learning (RL), where the controller and disturbance play a zero-sum dynamic game. Transitions collected through black-box interactions are stored in a replay buffer and used to jointly train a safety critic \(Q_\omega(x,u,d)\), a controller actor \(\pi^u_\theta(x)\), and a disturbance actor \(\pi^d_\psi(x,u)\).
The critic minimizes the Bellman residual of the time-discounted Isaacs equation. The controller maximizes the safety critic to learn a best-effort fallback policy, while the disturbance minimizes it to learn an adversarial response. This enables scalable synthesis without explicit knowledge of the system dynamics.
Training the worst-case best-response disturbance policy
The disturbance policy learned through adversarial RL is trained to provide a worst-case response to the equilibrium control policy. However, it does not necessarily minimize the safety critic for arbitrary control inputs generated by the \(Q\)-CBF optimization. This mismatch may be exploitable: a disturbance response learned against the fallback policy need not remain worst-case for the filtered policy.
We therefore further train a best-response disturbance policy \(\pi^d_{\tilde{\psi}}(x,u)\) against a diverse set of control policies. The disturbance actor is trained to minimize \(Q_\omega(x,u,d)\), using state–control pairs \((x,u)\) sampled from these policies. In practice, we collect controller checkpoints from multiple RL training runs with different random seeds, exposing the disturbance policy to a broad distribution of controller behaviors. This encourages \(\pi^d_{\tilde{\psi}}(x,u)\) to approximate a local minimizer of the critic across the state–control space.
Neural Q-CBF safety filter
@inproceedings{oh2026synthesis,
author = {Oh, Donggeon David and Nguyen, Duy P and Hu, Haimin and Fisac, Jaime Fern{\'a}ndez},
booktitle = {IEEE Conference on Decision and Control (CDC)},
organization = {IEEE},
title = {Synthesis and deployment of maximal robust control barrier functions through adversarial reinforcement learning},
year = {2026}
}
Authors
We consider a system with state \(x \in X\), control \(u \in U\), and bounded disturbance \(d \in D\). The dynamics are given by \(x_{t+1} = f(x_t, u_t, d_t)\).
The failure set \(F = \{x \in X \mid g(x) < 0\}\) contains unacceptable states, where \(g\) is a continuous safety margin function. Our goal is to prevent the system from entering \(F\) at all times, despite all admissible disturbances.
The maximal robust safe set \(\Omega^\star\) consists of all initial states from which a control policy can guarantee this objective:
Here, \(x_x^{\pi^u,\pi^d}(t)\) denotes the trajectory starting from \(x_0 = x\) under control policy \(\pi^u\) and disturbance policy \(\pi^d\).
The safety value function \(V\) is a solution to the dynamic programming safety Isaacs equation:
Its zero-superlevel set recovers the maximal robust safe set: \(\Omega^\star = \{x \in X \mid V(x) \geq 0\}\).
We also define the state–control–disturbance safety value function \(Q(x,u,d)\) as
Using the theorem above, we derive a robust discrete-time CBF safety filter that guarantees safety on the maximal robust safe set \(\Omega^\star\). At each timestep, the filter solves an optimal control problem to find the input closest to the task policy's proposal while satisfying the robust \(Q\)-CBF constraint.
Once \(Q\) and \(V\) are available, the state–control–disturbance lift of the safety value enables evaluation of the robust \(Q\)-CBF constraint from these value functions alone, without any knowledge of the system dynamics. In particular, no explicit closed-form dynamics, control-affine assumptions, or prescribed uncertainty structure are required.
At each timestep, the neural \(Q\)-CBF safety filter receives the current state \(x\) and the task input \(\pi^{\text{task}}(x)\). For each candidate control input \(u\), the learned best-response disturbance policy predicts \(\tilde{d} = \pi^d_{\tilde{\psi}}(x,u)\). The neural safety critic then evaluates \(Q_\omega(x,u,\tilde{d})\) as a tractable surrogate of the worst-case safety value:
This plug-in approximation removes the inner disturbance optimization: evaluating each candidate requires one forward pass through the disturbance policy network and one through the safety critic. The outer optimization selects the control input closest to the task input while requiring the surrogate safety value to satisfy the \(Q\)-CBF constraint, with \(V(x)\) also evaluated using its neural approximation.
Thus, constraint evaluation relies on the learned safety values and disturbance policy, without explicit knowledge of the system dynamics. This enables tractable runtime safety filtering for high-dimensional black-box systems.
We evaluate the framework on a disturbed inverted pendulum, with control input \(u \in [-20,20]\) and disturbance force \(F \in [-2,2]\). The state consists of the angle \(\theta\) from the upright equilibrium and angular velocity \(\dot{\theta}\). Failure occurs when \(|\theta| > \pi/3\).
We compare the zero-superlevel set of the neural \(Q\)-CBF (purple) with an analytically designed valid robust CBF (orange), a heuristic barrier candidate (gray), and the maximal robust safe set computed numerically using OptimizedDP (blue). The heuristic barrier candidate is not valid under the imposed control bound and is included only as a heuristic baseline.
All filters use the same PD task controller. To stress-test robustness, we numerically compute a best-response disturbance against each resulting closed-loop controller. For each filter, 20 trajectories are initialized near the boundary of its zero-superlevel set and simulated under these filter-specific worst-case disturbances.
All three filtering schemes achieve 100% empirical safety in this test. The neural \(Q\)-CBF is substantially less conservative than the barrier-based baselines, with a zero-superlevel set that nearly recovers the maximal robust safe set.
Pure pursuit policy; no filter
Experiments
Disturbed Inverted Pendulum
Pendulum experiment. The zero-superlevel sets of the neural \(Q\)-CBF (purple), valid robust CBF (orange), and heuristic barrier candidate (gray) are compared with the numerically computed maximal robust safe set (blue). Red dotted lines mark the failure boundaries at \(\theta = \pm \pi/3\). Trajectories are initialized near each filter’s zero-superlevel-set boundary and simulated under filter-specific worst-case disturbances. The neural \(Q\)-CBF nearly recovers the maximal robust safe set, substantially reducing conservatism relative to the barrier-based baselines.
Quadrupedal locomotion
Least-restrictive safety filter
Neural Q-CBF safety filter
Quadrupedal locomotion under adversarial disturbances. Left: Trajectories for the unfiltered policy without disturbance (dark gray), unfiltered policy under disturbance (light gray), LRSF (blue), and neural \(Q\)-CBF (purple). The star marks the starting point, and red crosses indicate failures. Insets show representative robot configurations and applied disturbance forces. Right: The distribution of per-step task input deviation, \(\|u^{\text{task}} - u^{\text{CBF}}\|^2\), illustrates the smaller interventions of the neural \(Q\)-CBF compared with LRSF.
We evaluate quadrupedal locomotion using a 36-dimensional Unitree Go2 simulation in MuJoCo, treating its dynamics as a black box. The control consists of 12 joint-position increments, \(u \in [-0.5,0.5]^{12}\). External disturbance forces can reach \(50\,\mathrm{N}\), with arbitrary directions and application points on the torso. Failure occurs if any monitored torso corner has less than \(0.10\,\mathrm{m}\) of ground clearance.
Adversarial RL trains the safety critic and fallback policy used for safety filtering. We further train the best-response disturbance policy against diverse control policies and use this same adversary for all disturbed comparisons. A pure-pursuit task controller commands forward locomotion.
Unfiltered task policy. Without safety filtering, the task controller is vulnerable to adversarial disturbances, achieving only a 16% safe rate.
Least-restrictive safety filter (LRSF). Last-minute interventions amplify the effects of neural approximation error, yielding a 38% safe rate. Frequent switching between the task and learned fallback policies causes abrupt chattering and obstructs forward progress.
Neural \(Q\)-CBF safety filter. The robot maintains stable forward locomotion and achieves 100% empirical safety across 50 randomized trials, with smaller modifications to task inputs.