ISAACS: Iterative Soft Adversarial Actor Critic for Safety
ISAACS (Iterative Soft Adversarial Actor-Critic for Safety) is a new game-theoretic reinforcement learning scheme for approximate safety analysis, whose simulation-trained control policies can be efficiently converted at runtime into robust safety-certified control strategies, allowing robots to plan and operate with safety guarantees in the physical world.
How does it work?
To harness the scalability of neural representations without renouncing the robust safety guarantees of model-based analysis, we propose Iterative Soft Adversarial Actor-Critic for Safety (ISAACS), a game-theoretic reinforcement learning scheme that approximates the HJI solution to a reachability game and learns a safety policy that can be used to construct a provably safe runtime control strategy.
The robot’s safety controller and the disturbance’s policies are trained together through a sequence of simulated “safety games”, in which the controller is trying to avoid catastrophic failures (such as falls or collisions) while the disturbance attempts to force these to happen. This adversarial disturbance agent can instantiate different possible realizations of the robot’s uncertainty about the real world (e.g. terrain, wind, actuator noise…). While the two actors learn to improve their strategies, we also train a critic whose role is to predict how close the robot will come to a safety violation from any initial state, as well as how much any candidate action by the robot or the disturbance would contribute to increasing or decreasing this future margin.
The learned control policy can then be treated as an “untrusted oracle” and used in online safety certification by guiding a robust predictive rollout that accounts for all admissible realizations of model uncertainty. Specifically, the learned controller generates a reference trajectory. Equipped with tracking policies, it computes forward reachable sets (FRSs) of tracking errors under known disturbance bound. The “safety filter” then checks if the footprint-augmented FRSs overlap with obstacles. This “rollout check” enables a recursively safe runtime control filter that preemptively overrides any candidate control action that could otherwise drive the state into an unrecoverable configuration.
Our proposed robust rollout-based safety filter is applied in a receding-horizon fashion. We always keep a series of fallback policies, which are updated whenever a new certification is constructed.
Results
ISAACS vs. SAC vs. DR
ISAACS is more robust to disturbance compared to uncertainty-agnostic soft actor-critic (SAC) and uniformly-sampled domain randomization (DR). In fact, it has similar safety performance as oracle solutions.
Confusion matrix of ISAACS vs. oracle
Top: learned safety critic can wrongly predict some rollout outcomes, leading to inaccuracies in the estimated safe set boundary.
Middle: learned ISAACS safety policy achieves near-optimal success but is occasionally suboptimal near the safe set boundary.
Bottom: direct policy rollout using the learned disturbance can lead to over-optimistic predictions.
Different Safety Filter Criteria
Our proposed robust rollout-based safety filter: achieves a perfect 100% safety rate for a long enough lookahead horizon (50 time steps),
A direct gameplay rollout safety filter, which rolls out the trajectory by the learned ISAACS control and disturbance policy, has a 99% safe rate. Additionally, using a value-based safety filter with the sign of the learned safety critic renders a 94% safe rate only.
Although the value-based safety filter can be improved by introducing a small threshold ϵ = 0.05 to mitigate approximation errors, this lacks safety guarantees. The threshold tunning is ad-hoc and difficult before real deployment.
Can ISAACS scale to high-dimensional dynamics? (work in progress)
Yes, we have applied ISAACS to 36-D locomotion experiments. Here the quadruped tries to prevent itself from falling over against forces from different directions. Stay tuned for more results!
@inproceedings{hsunguyen2023isaacs,
title={ISAACS: Iterative Soft Adversarial Actor-Critic for Safety},
author={Hsu, Kai-Chieh and Nguyen, Duy Phuong and Fisac, Jaime Fern\`andez},
booktitle={Proceedings of the 5th Annual Learning for Dynamics and Control Conference},
page={90—103}
year={2023},
editor={Matni, Nikolai and Morari, Manfred and Pappas, George J.},
volume={211},
series={Proceedings of Machine Learning Research},
month={15--16 Jun},
publisher={PMLR},
url={https://proceedings.mlr.press/v211/hsu23a.html}}
Citation
Authors
This work is supported in part by the Google Research Scholar Award. The authors thank Jie Tan and Wenhao Yu for their thoughtful suggestions and valuable insights.