How to Run the Notebook

  1. In your terminal:
docker compose up
  1. In your web browser, open localhost:8888
  2. Enter the token my-token
  3. Navigate into the work/ folder, and open the GridWorld notebook in the JupyterLab environment.

Link to Github repo for the implementation and notebook used to generate some figures:

https://github.com/shantanu/grid-world

Introduction

The most basic Dynamic Programming application is Gridworld.

image.png

We can represent this grid as a 4x4 array, and actions are {u, d, l, r} in each state. This is an undiscounted, episodic task. The states 0 and 15 (gray above) are terminal. If an action takes the agent off the grid, the state is unchanged but we still get -1 reward.

For the first phase of the project, I want to evaluate the equiprobable random policy, where in each state the agent has a 0.25 chance of going in any direction.

Policy Evaluation

Problem: given a policy $\pi$, I want to evaluate $V_\pi(s)$, the state value function, for all s.

This helps because once we have the value function, we can use it to do greedy policy improvement and policy iteration to find the optimal policy.

How: We can iterate the Bellman Equation in place: