Adversarial Inverse Reinforcement Learning

Adversarial Inverse Reinforcement Learning (AIRL)¹

goal: derive a reward function $r_{\theta }$ from a demonstration dataset of expert trajectories

method:

• solve the maximum likelihood problem $ma{x}_{\theta }{\mathbb{E}}_{\tau \sim \mathcal{D}}[\log p_{\theta }(\tau )]$, with $p_{\theta }(\tau )\propto p(s_0){\prod }_{t=0}^{T}p(s_{t+1}|s_t,a_t)e^{{\gamma }^{t}r\theta (s_t,a_t)}$
• specifically, for adversarial IRL the above maximization problem is cast as a GAN and into the single state and action case
  • A discriminator $D_{\theta }(s,a)=\frac{\exp \{f_{\theta }(s,a)\}}{\exp \{f_{\theta }(s,a)\}+\pi (a|s)}$ is trained using binary cross-entropy
  • An agent (=generator) maximizing its return using $r(s,a)=\log (D_{\theta }(s,a))-\log (1-D_{\theta }(s,a))$

Footnotes §

1. Coined in Learning robust rewards with Adversarial Inverse Reinforcement Learning (Fu et al., 2018) ↩