⌘K
ENES
GitHub
indexintermediateplannedReinforcement Learning#reinforcement-learning#decision-making#policies#rewards

Reinforcement Learning

Reinforcement learning is learning from consequences over time. The object being learned is not a label predictor but a policy or value estimate for sequential decisions under uncertainty.

Mental model

An agent observes a state, chooses an action, receives a reward, and transitions. Return aggregates future rewards; a policy maps states to action distributions; value functions summarize expected return. Bellman equations expose the recursive structure that most RL algorithms approximate.

Current overview

Candidate note roadmap

  • mdps-returns-policies-and-values — formal problem definition and assumptions.
  • bellman-equations-and-dynamic-programming — policy evaluation, improvement, and value iteration.
  • monte-carlo-and-temporal-difference-learning — sampled returns, bootstrapping, bias, and variance.
  • q-learning-and-deep-q-networks — off-policy control, replay, target networks, and instability.
  • policy-gradients-from-first-principles — score-function estimator, baselines, and variance reduction.
  • actor-critic-methods — learned critics, advantage estimation, and implementation tradeoffs.
  • exploration-and-credit-assignment — uncertainty, sparse rewards, and long horizons.
  • offline-rl-and-dataset-coverage — distribution shift, conservatism, and evaluation without deployment.
  • model-based-rl-and-world-models — learned dynamics, planning, model error, and compounding bias.
  • reward-hacking-rlhf-and-rlaif — specification gaming and connections to model alignment.

Scope

This branch separates RL as a field from RLHF as one adaptation technique. Examples start in tabular environments before neural approximators hide the mechanism.

Connects to: Classical AI and Reasoning · RLHF with PPO · Autonomy and Control

Core sources