Statistical Reinforcement Learning: Modern Machine Learning Approaches > 인공지능

Table of Contents

Introduction to Reinforcement Learning
Reinforcement Learning
Mathematical Formulation
Structure of the Book
Model-Free Policy Iteration
Model-Free Policy Search
Model-Based Reinforcement Learning

MODEL-FREE POLICY ITERATION

Policy Iteration with Value Function Approximation
Value Functions
State Value Functions
State-Action Value Functions
Least-Squares Policy Iteration
Immediate-Reward Regression
Algorithm
Regularization
Model Selection
Remarks

Basis Design for Value Function Approximation
Gaussian Kernels on Graphs
MDP-Induced Graph
Ordinary Gaussian Kernels
Geodesic Gaussian Kernels
Extension to Continuous State Spaces
Illustration
Setup
Geodesic Gaussian Kernels
Ordinary Gaussian Kernels
Graph-Laplacian Eigenbases
Diffusion Wavelets
Numerical Examples
Robot-Arm Control
Robot-Agent Navigation
Remarks

Sample Reuse in Policy Iteration
Formulation
Off-Policy Value Function Approximation
Episodic Importance Weighting
Per-Decision Importance Weighting
Adaptive Per-Decision Importance Weighting
Illustration
Automatic Selection of Flattening Parameter
Importance-Weighted Cross-Validation
Illustration
Sample-Reuse Policy Iteration
Algorithm
Illustration
Numerical Examples
Inverted Pendulum
Mountain Car
Remarks

Active Learning in Policy Iteration
Efficient Exploration with Active Learning
Problem Setup
Decomposition of Generalization Error
Estimation of Generalization Error
Designing Sampling Policies
Illustration
Active Policy Iteration
Sample-Reuse Policy Iteration with Active Learning
Illustration
Numerical Examples
Remarks

Robust Policy Iteration
Robustness and Reliability in Policy Iteration
Robustness
Reliability
Least Absolute Policy Iteration
Algorithm
Illustration
Properties
Numerical Examples
Possible Extensions
Huber Loss
Pinball Loss
Deadzone-Linear Loss
Chebyshev Approximation
Conditional Value-At-Risk
Remarks

MODEL-FREE POLICY SEARCH

Direct Policy Search by Gradient Ascent
Formulation
Gradient Approach
Gradient Ascent
Baseline Subtraction for Variance Reduction
Variance Analysis of Gradient Estimators
Natural Gradient Approach
Natural Gradient Ascent
Illustration
Application in Computer Graphics: Artist Agent
Sumie Paining
Design of States, Actions, and Immediate Rewards
Experimental Results
Remarks

Direct Policy Search by Expectation-Maximization
Expectation-Maximization Approach
Sample Reuse
Episodic Importance Weighting
Per-Decision Importance Weight
Adaptive Per-Decision Importance Weighting
Automatic Selection of Flattening Parameter
Reward-Weighted Regression with Sample Reuse
Numerical Examples
Remarks

Policy-Prior Search
Formulation
Policy Gradients with Parameter-Based Exploration
Policy-Prior Gradient Ascent
Baseline Subtraction for Variance Reduction
Variance Analysis of Gradient Estimators
Numerical Examples
Sample Reuse in Policy-Prior Search
Importance Weighting
Variance Reduction by Baseline Subtraction
Numerical Examples
Remarks

MODEL-BASED REINFORCEMENT LEARNING

Transition Model Estimation
Conditional Density Estimation
Regression-Based Approach
Q-Neighbor Kernel Density Estimation
Least-Squares Conditional Density Estimation
Model-Based Reinforcement Learning
Numerical Examples
Continuous Chain Walk
Humanoid Robot Control
Remarks

Dimensionality Reduction for Transition Model Estimation
Sufficient Dimensionality Reduction
Squared-Loss Conditional Entropy
Conditional Independence
Dimensionality Reduction with SCE
Relation to Squared-Loss Mutual Information
Numerical Examples
Artificial and Benchmark Datasets
Humanoid Robot
Remarks

References
Index