Reinforcement Learning

https://rail.eecs.berkeley.edu/deeprlcourse/

https://www.youtube.com/playlist?list=PLoROMvodv4rOSOPzutgyCTapiGlY2Nd8u

From https://qr.ae/prnRfl

In my opinion, the main RL problems are related to:

Information representation: from POMDP to predictive state representation to TD-networks to deep-learning. Inverse RL: how to learn the reward Algorithms

• Off-policy
• Large scale: linear and nonlinear approximations of the value function
• Policy search vs. Q-learning based

Beyond MDP, contextual MDPs Optimal exploration Hierarchical RL Multitask RL and life long learning Multiagent RL Efficient model based RL

I can suggest good papers for each of these problems, but there are few books. This is a list of most of the books I am aware of:

• Sutton & Barto Book: Reinforcement Learning: An Introduction. The classical intuitive intro to the field. I love it.
• The chapter by Bertsekas: Page on mit.edu
• Algorithms for Reinforcement Learning: Csaba Szepesvari. Nice compendium of ready to be implemented algorithms.
• Approximators. Busoniu, Lucian; Robert Babuska ; Bart De Schutter ; Damien Ernst (2010). This is a very practical book that explains some state-of-the-art algorithms (i.e., useful for real world problems) like fitted-Q-iteration and its variations.
• Reinforcement Learning: State-of-the-Art. Vol. 12 of Adaptation, Learning and Optimization. Wiering, M., van Otterlo, M. (Eds.), 2012. Springer, Berlin. In Sutton's words "This book is a valuable resource for students wanting to

go beyond the older textbooks and for researchers wanting to easily catch up with recent developments".

• Optimal Adaptive Control and Differential Games by Reinforcement Learning Principles : Draguna Vrabie, Kyriakos G. Vamvoudakis , Frank L. Lewis. I am not familiar with this one, but I have seen it recommended.
• Markov Decision Processes in Artificial Intelligence, Sigaud O. & Buffet O. editors, ISTE Ld., Wiley and Sons Inc, 2010.

I definitely suggest the books by Sutton and Barto as an excellent intro, the chapter by Bertsekas for getting a solid theoretical background and the book by Busoniu et al. for practical algorithms that can solve some non-toy problems. I also find useful the book by Szepesvari as a quick reference for understanding an comparing algorithms.

There are also several good specialized monographs and surveys on the topic, some of these are:

• "From Bandits to Monte-Carlo Tree Search: The Optimistic Principle Applied to Optimization and Planning" by Remi Munos (New trends on Machine Learning). This monograph covers important nonconvex optimistic optimization methods that can be applied for policy search. Available at https://hal.archives-ouvertes.fr/hal-00747575v5/document
• "Reinforcement Learning in Robotics: A Survey" by J. Kober, J. A. Bagnell and J. Peters. Available at Page on ias.tu-darmstadt.de
• "A Tutorial on Linear Function Approximators for Dynamic Programming and Reinforcement Learning" by A. Geramifard, T. J. Walsh, S. Tllex, G. Chowdhary, N. Roy and J. P. How (Foundations and Trends in Machine Learning). Available at Page on mit.edu
• "A Survey on Policy Search for Robotic" by Newmann and Peters (Foundations and Trends in Machine Learning). Available at Page on ausy.tu-darmstadt.de

• Reinforcement Learning Problem is the AI Problem
• Stable Baseline">Collection of RL Algorithms
• A collection of psuedocode of RL Algorithms with notes on which to use when
• An excellent collection of RL algorithms and their comparision

• 321 : \(\widehat{\nabla{J(\theta)}}\) is a stochastic estimate whose expectation approximates the gradient of the performance measure with respect to its argument.

  “Stochastic Estimate” triggered these thoughts:

  • We can not know the gradient of the performance, because it may be difficult to measure through out all states, or due to other reasons
  • In these cases, we can instead use “Stochastic Estimate” of the desired thing. This estimate has to approximate the true thing in its Expectation. ie. the Expectation (on average value) must be the desired thing.
  • Thus when we can’ t compute the thing we require, it may suffice to compute the thing which in ‘expectation’ is the desired thing.

• 329 Monte Carlo Method have high variance.

  As a Monte Carlo Method REINFORCE may be of high variance and thus produce slow learning.

• A (Long) Peek into Reinforcement Learning : https://lilianweng.github.io/posts/2018-02-19-rl-overview/

https://spinningup.openai.com/en/latest/user/introduction.html

https://andyljones.com/posts/rl-debugging.html talks about:

• Probe Environments
• Probe Agents
• Logs execessively
• Use really large batch size
• Rescale your rewards

TalkRL: https://podcasts.apple.com/us/podcast/talkrl-the-reinforcement-learning-podcast/id1478198107

• MARL
• https://github.com/thu-ml/tianshou
• https://pettingzoo.farama.org/api/aec/

Backlinks

• RL can be used to train Non-Differentiable Models
• Procgen Benchmark
• Genetic Algorithm
• Sergey Levine
• Supervised Learning for RL