Trjtdtk

Do I have to worry about players making “bad” choices on level up?

Stark VS Thanos

Are Boeing 737-800’s grounded?

What's the metal clinking sound at the end of credits in Avengers: Endgame?

Feels like I am getting dragged in office politics

Why does processed meat contain preservatives, while canned fish needs not?

Has any spacecraft ever had the ability to directly communicate with civilian air traffic control?

"ne paelici suspectaretur" (Tacitus)

Historically, were women trained for obligatory wars? Or did they serve some other military function?

What gives an electron its charge?

Why does nature favour the Laplacian?

Build a trail cart

Can fracking help reduce CO2?

Any examples of headwear for races with animal ears?

You look catfish vs You look like a catfish

Find the coordinate of two line segments that are perpendicular

How to determine the actual or "true" resolution of a digital photograph?

Single Colour Mastermind Problem

Where did the extra Pym particles come from in Endgame?

How can I get precisely a certain cubic cm by changing the following factors?

How does a Swashbuckler rogue "fight with two weapons while safely darting away"?

When and why did journal article titles become descriptive, rather than creatively allusive?

Sci-fi novel series with instant travel between planets through gates. A river runs through the gates

What's the polite way to say "I need to urinate"?

REINFORCE algorithm with discounted rewards – where does gamma^t in the update come from?

Reinforcement learning: understanding this derivation of n-step Tree Backup algorithmWhy do we normalize the discounted rewards when doing policy gradient reinforcement learning?How can we use the current rewards as a system input in the RUN time when working with Deep Q learning?Does self driving technology gain more from the data or the state-of-the-art algorithm?RL Policy Gradient: How to deal with rewards that are strictly positive?MDP - RL, Multiple rewards for the same state possible?Reinforcement learning: Discounting rewards in the REINFORCE algorithmHow does Q-Learning deal with mixed strategies?Pytorch: How to create an update rule the doesn't come from derivatives?

1

$begingroup$

I'm looking at Sutton & Barto's rendition of the REINFORCE algorithm (from their book here, pg. 328).

I can't quite understand why there is $gamma^t$ on the last line. They say:

[..] in the boxed algorithms we are giving the algorithms for the general discounted [return] case. All of the ideas go through in the discounted case with appropriate adjustments [..] but involve additional complexity that distracts from the main ideas.

This doesn't quite make it clear for me.

I didn't find any other description of REINFORCE that would include this. Does anyone have an idea how the $gamma^t$ gets there?

reinforcement-learning policy-gradients

share|improve this question

asked Apr 8 at 10:20

TuetschekTuetschek

61

$endgroup$

• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  Later rewards should be less important than current.
  $endgroup$
  – Carl Rynegardh
  Apr 8 at 12:58
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  Thanks, I think you're right – or as a friend of mine put it, samples that have less following time steps should be less important. So the intuition is there, but I'll leave this here in case someone is able to do the math :-).
  $endgroup$
  – Tuetschek
  Apr 9 at 10:32
  

  
  
  
  

add a comment | 

1

$begingroup$

I'm looking at Sutton & Barto's rendition of the REINFORCE algorithm (from their book here, pg. 328).

I can't quite understand why there is $gamma^t$ on the last line. They say:

[..] in the boxed algorithms we are giving the algorithms for the general discounted [return] case. All of the ideas go through in the discounted case with appropriate adjustments [..] but involve additional complexity that distracts from the main ideas.

This doesn't quite make it clear for me.

I didn't find any other description of REINFORCE that would include this. Does anyone have an idea how the $gamma^t$ gets there?

reinforcement-learning policy-gradients

share|improve this question

asked Apr 8 at 10:20

TuetschekTuetschek

61

$endgroup$

• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  Later rewards should be less important than current.
  $endgroup$
  – Carl Rynegardh
  Apr 8 at 12:58
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  Thanks, I think you're right – or as a friend of mine put it, samples that have less following time steps should be less important. So the intuition is there, but I'll leave this here in case someone is able to do the math :-).
  $endgroup$
  – Tuetschek
  Apr 9 at 10:32
  

  
  
  
  

add a comment | 

$begingroup$

I'm looking at Sutton & Barto's rendition of the REINFORCE algorithm (from their book here, pg. 328).

I can't quite understand why there is $gamma^t$ on the last line. They say:

[..] in the boxed algorithms we are giving the algorithms for the general discounted [return] case. All of the ideas go through in the discounted case with appropriate adjustments [..] but involve additional complexity that distracts from the main ideas.

This doesn't quite make it clear for me.

I didn't find any other description of REINFORCE that would include this. Does anyone have an idea how the $gamma^t$ gets there?

reinforcement-learning policy-gradients

share|improve this question

asked Apr 8 at 10:20

TuetschekTuetschek

61

$endgroup$

share|improve this question

asked Apr 8 at 10:20

TuetschekTuetschek

61

share|improve this question

asked Apr 8 at 10:20

TuetschekTuetschek

61

asked Apr 8 at 10:20

TuetschekTuetschek

61

asked Apr 8 at 10:20

TuetschekTuetschek

61