Trjtdtk

In RL Course by David Silver - Lecture 7: Policy Gradient Methods, David explains what an Advantage function is, and how it's the difference between Q(s,a) and the V(s)

Preliminary, from this post:

First recall that a policy $pi$ is a mapping from each state, $s$, action $a$, to the probability $pi(a mid s)$ of taking action $a$ when in state $s$.

The state value function, $V^pi(s)$, is the expected return when starting in state $s$ and following $pi$ thereafter.

Similarly, the state-action value function, $Q^pi(s, a)$, is the expected return of when starting in state $s$, taking action $a$, and following policy $pi$ thereafter.

In my understanding, $V(s)$ is always larger than $Q(s, a)$, because the function $V$ includes the reward for the current state $s$, unlike $Q$. So, why is the advantage function defined as $A = V - Q$ rather than $A = Q - V$ (at minute 1:12:29 in the video)?

Actually, V might not be larger than Q, because $s$ might actually contain a negative reward. In such a case how can we be certain what to subtract from what, such that our Advantage is always positive?

$Q(s, a)$ returns a value of entire total reward that's expected ultimately, after we pick an action $a$. $V(s)$ is the same, just with an extra reward from current state $s$ as well.

I don't see why a value of $Q - V$ would be useful. On the other hand, $V - Q$ would be useful because it would tell us the reward we would get on $s_t+1$ if we took the action $a$.

In my understanding, $V(s)$ is always larger than $Q(s,a)$, because the function $V$ includes the reward for the current state $s$, unlike $Q$

This is incorrect. There is not really such a thing as "the reward for current state" in the general case of a MDP. If you mean the $V(S_t)$ should include the value of $R_t$, then this is still wrong, given David Silver's use of the conventions for time step indexing. It is possible to associate immediate reward with either the current time step, leading to sequence $S_0, A_0, R_0, S_1, A_1, R_1$ etc or you can use the convention of immediate reward being on next time step $S_0, A_0, R_1, S_1, A_1, R_2$ etc. David Silver (and Sutton & Barto's book) uses the latter convention.

Under that convention:

$$V(s) = mathbbE_pi[sum_k=0^infty
gamma^kR_t+k+1|S_t=s]$$

$$Q(s,a) = mathbbE_pi[sum_k=0^infty gamma^kR_t+k+1|S_t=s, A_t=a]$$

You can see that the first term in the expansion of the sum for both Q(s,a) and V(s) is $R_t+1$. If you changed the convention, then both would include the equivalent value, but would be labelled $R_t$ in any formula.

Q and V do not differ in which time steps they sum reward over. They may differ in the value of $R_t+1$ because $V(s)$ assumes following the policy $pi$ when selecting $A_t$ whilst $Q(s,a)$ uses the value $a$ supplied as a parameter for $A_t$, which can be different.

how can we be certain what to subtract from what, such that our Advantage is always positive?

Advantage can be negative, that is fine. It means that the action $a$ in $A(s,a)$ is a worse choice than the current policy's.

I don't see why a value of $(Q−V)$ would be useful. On the other hand, $(V−Q)$ would be useful

Both would be equally useful, it is mainly convention that we work with finding maximum Advantage representing the benefit of choosing a specific action instead of following the current policy, as opposed to finding the minimum "Disadvantage". However, the concept of Advantage in this context is arguably the more natural view.

because it would tell us the reward we would get on $s_t+1$ if we took the action $a$

As explained above, this is wrong. The value in $A(s,a)$ expresses the potential benefit we would get for changing the policy $pi(s)$. That might include changes to $R_t+1$, but is not limited to a single time step.

Some RL approaches do create a predictive function for expected immediate reward $hatr(s,a)$ - typically this is a secondary component, used to help refine parameters for other function approximators.