Reinforcement Learning

Papers

• Deep reinforcement learning from human preferences
• Direct Preference Optimization Your Language Model is Secretly a Reward Model - DPO
• Group Robust Preference Optimization in Reward-free RLHF - GRPO
• Illustrating Reinforcement Learning from Human Feedback (RLHF)
• Is Feedback All You Need Leveraging Natural Language Feedback in Goal-Conditioned Reinforcement Learning
• Learning to summarize from human feedback
• Playing Atari with Deep Reinforcement Learning
• Scaling Laws for Reward Model Overoptimization
• Training language models to follow instructions with human feedback - InstructGPT

Surveys and Reviews

• LLM Post-Training A Deep Dive into Reasoning Large Language Models

Implementation

• OpenRLHF docs - not very clean or readable according to Duarte; Patrick says Ray is a pain in the ass (authors present at RayCon so def. using Ray)
  • OpenRLHF An Easy-to-use, Scalable and High-performance RLHF Framework
• Gymnasium Documentation and Gymnasium repo - An API standard for single-agent reinforcement learning environments, with popular reference environments and related utilities (formerly Gym)
  • a maintained fork of OpenAI’s Gym library

Resources

• Reinforcement Learning from Human Feedback: A short introduction to RLHF and post-training focused on language models. by Nathan Lambert
• Reinforcement Learning An Overview - RL tutorial by Kevin Murphy
• Reinforcement Learning (Data Driven Science & Engineering - Chapter 11) by Steven L. Brunton and J. Nathan Kutz
  • link to PDF on Drive
• Foundations of Deep RL lecture series by Pieter Abbeel
• Policy Gradient Algorithms - post by Lilian Weng
• ✨ Reinforcement Learning Lecture Series by by Steve Brunton - follows the book chapter above
  • Overview of Deep Reinforcement Learning Methods
  • Model Based Reinforcement Learning: Policy Iteration, Value Iteration, and Dynamic Programming
  • Q-Learning: Model Free Reinforcement Learning and Temporal Difference Learning
• DeepMind x UCL | Introduction to Reinforcement Learning 2015 - 10 instalment lecture series by the one and only David Silver
  • Lecture 1: Introduction to Reinforcement Learning
  • Lecture 2: Markov Decision Process
• Reinforcement Learning: An Introduction (2015) by Richard S. Sutton and Andrew G. Barto
  • Webpage
  • References (PDFs)
• Multi-Agent Reinforcement Learning: Foundations and Modern Approaches (2024) by Stefano V. Albrecht,  Filippos Christianos,  Lukas Schäfer. MIT Press
• OpenAI Spinning Up
• MIT 6.S091: Introduction to Deep Reinforcement Learning (Deep RL) by Lex Fridman - link
• Deep Reinforcement Learning Course - HF 🤗 course
  • backed by the Hugging Face Deep Reinforcement Learning Course repo
  • chapter 1 “clipped” is from an older version: An Introduction to Deep Reinforcement Learning
• An Intuitive Explanation of Policy Gradient — Part 1 REINFORCE
• Posts by Nathan Lambert on (i.e. tagged with) RLHF: https://www.interconnects.ai/t/rlhf

# Reinforcement Learning from Human Feedback by Nathan Lambert

A short introduction to RLHF and post-training focused on language models. 👉 https://rlhfbook.com/

• Introductions
  1. Introduction
  2. Seminal (Recent) Works
  3. Definitions
• Problem Setup & Context
  1. Training Overview
  2. What are preferences?
  3. Preference Data
• Optimization Tools
  1. Reward Modeling
  2. Regularization
  3. Instruction Tuning
  4. Rejection Sampling
  5. Policy Gradients
  6. Direct Alignment Algorithms
• Advanced
  1. Constitutional AI and AI Feedback
  2. Reasoning and Reinforcement Finetuning
  3. Synthetic Data
  4. Evaluation
• Open Questions
  1. Over-optimization
  2. Style & Information
  3. Product, UX, Character, and Post-Training

---

See also

• Hamilton–Jacobi–Bellman equation
• Richard E. Bellman
• Bellman equation

---