Study notes: Learning About What You Do Not Do

A target policy \pi (the behaviour the question is about), a termination function \beta (when to stop counting), a cumulant r (the signal summed along the way) and an outcome z (the signal read at stopping). For example: “If I keep editing and rerunning the suite until it passes or I give up, how many of the twenty tests will pass?” Asked once per test, that is twenty questions. The world’s dynamics fix the answer, so the knowledge has a correctness condition the learner does not supply.

Off-policy learning, function approximation and bootstrapping. Ordinary TD iterates: it steps toward \Pi T^\lambda V_\theta and repeats, which converges only if that composition is a contraction. On-policy it is. Off-policy the projection is taken under the behaviour’s state distribution and the composition can stretch distances. Gradient TD descends the projected Bellman error, which is smallest at the fixed point whether or not the operator contracts. GQ(λ)’s secondary weights h estimate one factor of a product of expectations, so each step needs one sample.

The true answer is 0.9\times6=5.4.

Whole episodes (\lambda=1): approaching has ratio 1/0.5=2 and opening 1/0.25=4; the detour and giving up have ratio 0. One episode in eight approaches and opens and counts as 2\cdot4\cdot5.4=43.2. The average is 5.4, with a standard deviation of 14.3 per episode.

Bootstrap (\lambda=0): the door visits that opened give Q(D,\text{open})=6. The target for approaching is 0+0.9\sum_a\pi(D,a)Q(D,a)=5.4, with no ratio: the target policy enters through the expectation at the next state.

If the behaviour never approaches, nothing at the door is observed and no method recovers the 6.

Momentum carries each step’s update forward: \sum_{k\le t}\mu^{t-k}\delta_k\phi_k, every gradient paired with its own error. A trace multiplies old gradients by the current error: \delta_t\sum_{k\le t}\lambda^{t-k}\phi_k, so each gradient meets the decayed sum of later errors. Momentum changes how fast a solution is reached. \lambda changes which solution is reached.

The variance is e^{\sigma^2T}-1: e^2-1\approx6.4 at 200 tokens and e^{20}\approx5\times10^8 at 2{,}000. A language model computes the probability of every token it emits, so the ratio between two versions is exact, where a robot’s must be logged or modelled. The remedy is a state-dependent \lambda_t set low where the policies diverge, which cuts the trace and bootstraps from the current prediction. Retrace does this by truncating each ratio at 1.

\psi^\pi(s,a)=\mathbb E_\pi[\sum_t\gamma^tc_t\mid s,a], one general value function per cumulant, so Q^\pi_w=w^\top\psi^\pi and a new task costs one dot product. GPI acts by \arg\max_a\max_i w^\top\psi^{\pi_i}(s,a) over a library of policies. It is at least as good as every library policy on task w, up to the error in the learned \psi’s, with no learning for the new task. The set-max policy picks one library policy at the initial state and follows it to the end. GPI can switch from state to state and is never worse.

A cumulant; a policy in the library; that policy’s successor features at the start state; the set-max value; training the next policy against the library’s weakest weighting; GPI’s max at deployment. VPO averages over weightings where Zahavy’s iteration takes the worst case. At the horizon of one answer the tests compute each reward vector exactly, and GPI coincides with set-max because there is no later state at which to switch.

The first specialist is best for u\ge0.6: \bar w_1=\int_{0.6}^{1}(u,\,1-u)\,du=(0.32,\,0.08). By symmetry \bar w_2=(0.08,\,0.32). The balanced answer holds 0.4\le u\le0.6: \bar w_3=(0.10,\,0.10). They sum to (0.5,0.5), the weighting a scalar method would apply to all three. The max splits it by territory.

In the successor-feature construction the policy never sees w. Conditioning on w happens in the critic, as one dot product, and GPI selects among unconditioned behaviours. On the preference dataset the five criteria mostly agreed, so five cumulants spanned about one direction and a library of one policy covered every weighting. The rank of the reward vectors measures how large a library a task can use.

Exciting means b(s,a)>0 everywhere. Where the behaviour never acts as the target would, \rho_t=0, the trace is cut and nothing is learned; where it rarely does, the ratio is too large to use. A collapsed model produces no data about its lost alternatives, so no critic, later improvement step or self-distillation trained on its data can learn about them. A reward-diverse model is a better behaviour policy for whatever learning comes next. This is plasticity at the level of behaviour: weights that can still change, attached to a policy that no longer visits the states from which change would be learned.

Horde predicts: it learns off-policy what many behaviours would achieve, which is knowledge without behaviour. VPO behaves diversely: it keeps competent alternatives alive with no critic, which is behaviour without knowledge. GPI joins them: a library plus successor features meets a new weighting by a max. Both stop at the questions. Someone posed each general value function and wrote down each reward component. The world supplies the answers and not the questions.

Per-answer credit from the marginal gain each answer adds to the set, with no critic. A vector critic: linear GQ(λ) heads on frozen hidden states, one per reward component, giving a vector form of the generalized advantage. Continual VPO under drifting weights, with the prediction that a VPO checkpoint recovers best-of-k performance in fewer updates than a GRPO checkpoint. The caution: an answer kept by a max has no known behaviour policy, so off-policy correction applies to the raw samples and not to the survivors.