Study notes: How Can a Reward Train a Neural Network?
Recall questions for How Can a Reward Train a Neural Network?. Answer each from memory, then open the answer to compare. The link under a question leads to the section it draws on.
Questions
The delivery problem, used by several questions: at the junction J the detour ends the episode with reward 2; approaching the door gives 0 and leads to D; at the door, opening gives 6 and giving up gives 0; \gamma=0.9.
Evaluate a half-hearted policy
The policy approaches the door half the time and opens it a quarter of the time. Compute V^\pi(D), Q^\pi(J,\text{door}) and V^\pi(J).
Answer
V^\pi(D)=0.25\times6=1.5. Q^\pi(J,\text{door})=0+0.9\times1.5=1.35. V^\pi(J)=0.5\times2+0.5\times1.35=1.675. The door route holds the best return available, 5.4, and is still the worse choice under this behavior at the door.
Why one greedy step is not enough
Improve the policy above greedily, once. Which route does the new policy take, and why does it miss the best one? What fixes it?
Answer
The new policy opens at the door (6>0) and takes the detour (2>1.35), earning 2. The value 1.35 assumed the old behavior at the door. Re-evaluating gives V(D)=6 and Q(J,\text{door})=5.4, and a second greedy step takes the door. Evaluate, improve, repeat is policy iteration: a better later decision changes which earlier decision is right.
Where the max sits
In Q^*(s,a)=\mathbb E[r+\gamma\max_b Q^*(s',b)\mid s,a], why is the max inside the expectation?
Answer
The agent chooses actions; it does not choose which random outcome the environment produces. Its action value must average over that randomness. At the next state it can respond to what actually happened, so the max is taken there, after the outcome is known.
Two parameters, two jobs
What does \gamma set, and what does \lambda set?
Answer
\gamma is part of the objective: how much later rewards count. \lambda is part of the estimator: how observed rewards and value predictions are mixed to estimate that objective. Every target in the \lambda mixture estimates the same \gamma-discounted return. Lowering \lambda leans on the value estimates (bias) instead of long sampled outcomes (variance).
A gradient without the world’s derivative
A sampled reward has no label for the correct action, and the environment cannot be differentiated. Derive an estimator of \nabla_\theta\,\mathbb E[r] that needs neither.
Answer
Use \nabla_\theta\pi_\theta(a)=\pi_\theta(a)\nabla_\theta\log\pi_\theta(a). Then \nabla_\theta\sum_a\pi_\theta(a)\bar r(a)=\mathbb E_{a\sim\pi_\theta}[r\,\nabla_\theta\log\pi_\theta(a)], so the sampled reward times the gradient of the sampled action’s log-probability is an unbiased estimate. For a whole episode the log-probability of the trajectory is a sum, and the environment’s terms do not depend on \theta, so only the actions’ log-probabilities remain. The environment still decides which trajectories are sampled.
One sample at the door
At the door, opening has probability p=\sigma(\theta)=0.25. Give the two possible single-sample gradient estimates and their mean. Repeat with the baseline 1.5.
Answer
The log-probability derivatives are 1-p=0.75 for opening and -p=-0.25 for giving up. Without a baseline the estimates are 6\times0.75=4.5 and 0, with mean 0.25\times4.5=1.125=6p(1-p). With the baseline they are 4.5\times0.75=3.375 and (-1.5)\times(-0.25)=0.375, with the same mean. Both outcomes now push toward opening, and the two estimates are closer together.
Why a baseline is free
Show that subtracting b(s) from the return leaves the expected gradient unchanged.
Answer
\sum_a\pi_\theta(a\mid s)\nabla_\theta\log\pi_\theta(a\mid s)=\nabla_\theta\sum_a\pi_\theta(a\mid s)=\nabla_\theta 1=0. A baseline that does not depend on the sampled action multiplies this zero. Using \hat V(s) as the baseline turns the weight into an estimate of the advantage Q^\pi(s,a)-V^\pi(s).
Stored, replayed, off-policy, offline
Which of these make learning off-policy: storing a trajectory to compute its targets, a replay buffer, a fixed dataset, reusing a batch after one policy update?
Answer
Off-policy means learning about a policy other than the one that generated the data. Storing a trajectory does not do that by itself. A replay buffer usually does, because it holds data from older policies. Offline RL learns from a fixed dataset with no new collection, so it is off-policy for any policy that differs from the dataset’s. Reusing an on-policy batch after the update is off-policy, and the unbiased-gradient reading no longer holds.
What no estimator recovers
The robot always takes the detour. What can it learn about the door?
Answer
Nothing from its own data: no example reveals the reward of 6. A network may generalize from related situations, but its prediction about an untried action can be wrong. Exploration decides which predictions get tested. Part 2 of the continual-learning series returns to this as a coverage condition.
Terms
Introduced in this essay:
- policy
- Markov state
- return and discount factor
- state value and action value
- Bellman expectation equation
- policy iteration
- Bellman optimality equations
- temporal-difference error
- bootstrapping
- λ-return and eligibility trace
- on-policy and off-policy
- REINFORCE
- baseline and advantage
- generalized advantage estimation
- GRPO