WEEK 13: REINFORCEMENT LEARNING

Introduction to Reinforcement learning (RL), Q learning, Optimal portfolio choice application

December 08, 2025

← All Weeks

Weekly Materials

🎓

Lecture slides, week 13

Professor's lecture slides (PDF)
👩‍💻

TA Practice Slides

Hands-on tutorials and practice exercises

→
💻

Code Examples

Introduction to Reinforcement learning (RL) A notebook on RL basics A notebook on Q learning RL: Optimal portfolio choice application
📚

References & Resources

RL Materials Discussion of the previous exercises (with TA)

Additional Notes

Week 13: Reinforcement Learning

Learning Objectives

  • Understand the fundamentals of reinforcement learning (RL)
  • Master Q-learning algorithm and its applications
  • Explore the optimal portfolio choice problem using RL
  • Learn about Markov Decision Processes (MDPs)
  • Apply RL techniques to financial and decision-making problems
  • Understand the exploration-exploitation tradeoff

Topics Covered

  • Introduction to Reinforcement Learning: Agents, environments, rewards
  • Q-Learning: Model-free reinforcement learning algorithm
  • Optimal Portfolio Choice: Financial application of RL methods
  • Markov Decision Processes: Mathematical framework for RL
  • Policy vs Value Methods: Different approaches to RL
  • Exploration-Exploitation: Balancing learning and performance

Schedule

  • Lecture: Monday, December 8, 2025 (10:15 - 12:00)
  • Practice Session: Monday, December 8, 2025 (16:30 - 18:00)
  • TA Session: Discussion of RL implementations and capstone projects

Key Concepts

  • Agent-Environment Interaction: The RL loop
  • States, Actions, Rewards: Core RL components
  • Policy: Strategy for action selection
  • Value Functions: State-value and action-value functions
  • Temporal Difference Learning: Learning from experience
  • Bellman Equation: Fundamental recursive relationship

Reinforcement Learning Algorithms

  • Q-Learning: Off-policy temporal difference learning
  • SARSA: On-policy temporal difference learning
  • Policy Gradient Methods: Direct policy optimization
  • Deep Q-Networks (DQN): Deep learning meets RL
  • Actor-Critic Methods: Combining value and policy methods

Financial Applications

  • Portfolio Optimization: Using RL for asset allocation
  • Markowitz Framework: Modern portfolio theory with RL
  • Risk Management: Dynamic hedging strategies
  • Trading Strategies: Algorithmic trading using RL
  • Market Making: Optimal bid-ask spread strategies

Practical Implementation

  • RL Basics: Fundamental concepts and simple environments
  • Q-Learning Implementation: Step-by-step algorithm development
  • Financial Modeling: Least squares interpolation and Markowitz optimization
  • Performance Evaluation: Metrics for RL algorithm assessment

Real-world Examples

  • Game Playing: RL in strategic games
  • Robotics: Control and navigation applications
  • Recommendation Systems: Personalization using RL
  • Autonomous Systems: Decision making under uncertainty

Assignment

No new assignment this week - focus on capstone project development and RL exploration

Tools and Libraries

  • OpenAI Gym for RL environments
  • Custom financial environment implementations
  • NumPy and scikit-learn for basic RL algorithms
  • TensorFlow/PyTorch for deep RL (advanced)