How to write out the definition of the value function for continous action and state spaceWhat is the Q function and what is the V function in reinforcement learning?Reward dependent on (state, action) versus (state, action, successor state)Cannot see what the “notation abuse” is, mentioned by author of bookWhat is the difference between “expected return” and “expected reward” in the context of RL?How is that possible that a reward function depends both on the next state and an action from current state?Need help in deriving Policy Evaluation (Prediction)How is Importance-Sampling Used in Off-Policy Monte Carlo Prediction?Importance Sampling in Off-policy n-step SarsaMDP - RL, Multiple rewards for the same state possible?Evaluating value functions in RL
Turning a hard to access nut?
Deletion of copy-ctor & copy-assignment - public, private or protected?
How is the partial sum of a geometric sequence calculated?
Worshiping one God at a time?
If "dar" means "to give", what does "daros" mean?
Help prove this basic trig identity please!
How to generate binary array whose elements with values 1 are randomly drawn
In Aliens, how many people were on LV-426 before the Marines arrived?
Am I eligible for the Eurail Youth pass? I am 27.5 years old
Hausdorff dimension of the boundary of fibres of Lipschitz maps
Help rendering a complicated sum/product formula
What does "Four-F." mean?
How to define limit operations in general topological spaces? Are nets able to do this?
I seem to dance, I am not a dancer. Who am I?
Violin - Can double stops be played when the strings are not next to each other?
What does Jesus mean regarding "Raca," and "you fool?" - is he contrasting them?
World War I as a war of liberals against authoritarians?
What are substitutions for coconut in curry?
HP P840 HDD RAID 5 many strange drive failures
Practical application of matrices and determinants
Calculate the frequency of characters in a string
What exactly term 'companion plants' means?
Why is indicated airspeed rather than ground speed used during the takeoff roll?
Could Sinn Fein swing any Brexit vote in Parliament?
How to write out the definition of the value function for continous action and state space
What is the Q function and what is the V function in reinforcement learning?Reward dependent on (state, action) versus (state, action, successor state)Cannot see what the “notation abuse” is, mentioned by author of bookWhat is the difference between “expected return” and “expected reward” in the context of RL?How is that possible that a reward function depends both on the next state and an action from current state?Need help in deriving Policy Evaluation (Prediction)How is Importance-Sampling Used in Off-Policy Monte Carlo Prediction?Importance Sampling in Off-policy n-step SarsaMDP - RL, Multiple rewards for the same state possible?Evaluating value functions in RL
$begingroup$
In the book of Sutton and Barto (2018) Reinforcement Learning: An Introduction. The author defines the value function as.
$$v_pi(boldsymbols)=mathbbE_boldsymbola,sim, pileft[sum_k=0^inftygamma^kR_t+k+1,bigg|,boldsymbols_t=boldsymbols right]$$
If $boldsymbolain mathcalA$ and $boldsymbolsin mathcalS$ are continuous I would think by using Bellman's equation for the state-value function that this can be written as the integral
$$v_pi(boldsymbols)=int_boldsymbolainmathcalApileft(boldsymbola|boldsymbols right)int_boldsymbols'in mathcalSp(boldsymbols'|boldsymbols,boldsymbola)left[R_t+1+gamma v_pi(boldsymbols')right]dboldsymbols'dboldsymbola.$$
Is this correct?
Also without using Bellman's equation does the integral definition of the state-value function look like this?
$$v_pi(boldsymbols)=int_boldsymbolainmathcalApileft(boldsymbola|boldsymbols right)int_boldsymbols'in mathcalSp(boldsymbols'|boldsymbols,boldsymbola)left[R_t+1+gamma left[int_boldsymbola'inmathcalApileft(boldsymbola'|boldsymbols' right)int_boldsymbols''in mathcalSp(boldsymbols''|boldsymbols',boldsymbola')left[R_t+2+gammaleft[cdotsright] right]dboldsymbols''dboldsymbola'right] right]dboldsymbols'dboldsymbola$$
Are my integral versions correct?
reinforcement-learning
asked yesterday
MachineLearnerMachineLearner
1438
New contributor
MachineLearner is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
$endgroup$
add a comment |
$begingroup$
In the book of Sutton and Barto (2018) Reinforcement Learning: An Introduction. The author defines the value function as.
$$v_pi(boldsymbols)=mathbbE_boldsymbola,sim, pileft[sum_k=0^inftygamma^kR_t+k+1,bigg|,boldsymbols_t=boldsymbols right]$$
If $boldsymbolain mathcalA$ and $boldsymbolsin mathcalS$ are continuous I would think by using Bellman's equation for the state-value function that this can be written as the integral
$$v_pi(boldsymbols)=int_boldsymbolainmathcalApileft(boldsymbola|boldsymbols right)int_boldsymbols'in mathcalSp(boldsymbols'|boldsymbols,boldsymbola)left[R_t+1+gamma v_pi(boldsymbols')right]dboldsymbols'dboldsymbola.$$
Is this correct?
Also without using Bellman's equation does the integral definition of the state-value function look like this?
$$v_pi(boldsymbols)=int_boldsymbolainmathcalApileft(boldsymbola|boldsymbols right)int_boldsymbols'in mathcalSp(boldsymbols'|boldsymbols,boldsymbola)left[R_t+1+gamma left[int_boldsymbola'inmathcalApileft(boldsymbola'|boldsymbols' right)int_boldsymbols''in mathcalSp(boldsymbols''|boldsymbols',boldsymbola')left[R_t+2+gammaleft[cdotsright] right]dboldsymbols''dboldsymbola'right] right]dboldsymbols'dboldsymbola$$
Are my integral versions correct?
reinforcement-learning
asked yesterday
MachineLearnerMachineLearner
1438
New contributor
MachineLearner is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
$endgroup$
add a comment |
$begingroup$
In the book of Sutton and Barto (2018) Reinforcement Learning: An Introduction. The author defines the value function as.
$$v_pi(boldsymbols)=mathbbE_boldsymbola,sim, pileft[sum_k=0^inftygamma^kR_t+k+1,bigg|,boldsymbols_t=boldsymbols right]$$
If $boldsymbolain mathcalA$ and $boldsymbolsin mathcalS$ are continuous I would think by using Bellman's equation for the state-value function that this can be written as the integral
$$v_pi(boldsymbols)=int_boldsymbolainmathcalApileft(boldsymbola|boldsymbols right)int_boldsymbols'in mathcalSp(boldsymbols'|boldsymbols,boldsymbola)left[R_t+1+gamma v_pi(boldsymbols')right]dboldsymbols'dboldsymbola.$$
Is this correct?
Also without using Bellman's equation does the integral definition of the state-value function look like this?
$$v_pi(boldsymbols)=int_boldsymbolainmathcalApileft(boldsymbola|boldsymbols right)int_boldsymbols'in mathcalSp(boldsymbols'|boldsymbols,boldsymbola)left[R_t+1+gamma left[int_boldsymbola'inmathcalApileft(boldsymbola'|boldsymbols' right)int_boldsymbols''in mathcalSp(boldsymbols''|boldsymbols',boldsymbola')left[R_t+2+gammaleft[cdotsright] right]dboldsymbols''dboldsymbola'right] right]dboldsymbols'dboldsymbola$$
Are my integral versions correct?
reinforcement-learning
asked yesterday
MachineLearnerMachineLearner
1438
New contributor
MachineLearner is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
$endgroup$
In the book of Sutton and Barto (2018) Reinforcement Learning: An Introduction. The author defines the value function as.
$$v_pi(boldsymbols)=mathbbE_boldsymbola,sim, pileft[sum_k=0^inftygamma^kR_t+k+1,bigg|,boldsymbols_t=boldsymbols right]$$
If $boldsymbolain mathcalA$ and $boldsymbolsin mathcalS$ are continuous I would think by using Bellman's equation for the state-value function that this can be written as the integral
$$v_pi(boldsymbols)=int_boldsymbolainmathcalApileft(boldsymbola|boldsymbols right)int_boldsymbols'in mathcalSp(boldsymbols'|boldsymbols,boldsymbola)left[R_t+1+gamma v_pi(boldsymbols')right]dboldsymbols'dboldsymbola.$$
Is this correct?
Also without using Bellman's equation does the integral definition of the state-value function look like this?
$$v_pi(boldsymbols)=int_boldsymbolainmathcalApileft(boldsymbola|boldsymbols right)int_boldsymbols'in mathcalSp(boldsymbols'|boldsymbols,boldsymbola)left[R_t+1+gamma left[int_boldsymbola'inmathcalApileft(boldsymbola'|boldsymbols' right)int_boldsymbols''in mathcalSp(boldsymbols''|boldsymbols',boldsymbola')left[R_t+2+gammaleft[cdotsright] right]dboldsymbols''dboldsymbola'right] right]dboldsymbols'dboldsymbola$$
Are my integral versions correct?
reinforcement-learning
reinforcement-learning
asked yesterday
MachineLearnerMachineLearner
1438
New contributor
MachineLearner is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
asked yesterday
MachineLearnerMachineLearner
1438
New contributor
MachineLearner is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
asked yesterday
MachineLearnerMachineLearner
1438
New contributor
MachineLearner is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
asked yesterday
MachineLearnerMachineLearner
1438
asked yesterday
MachineLearnerMachineLearner
1438
1438
New contributor
MachineLearner is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
New contributor
MachineLearner is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
MachineLearner is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
$begingroup$
this can be written as the integral, is this correct?
Yes. Your derivations imply that we have assumed a deterministic reward given current state-action $(boldsymbols,boldsymbola)$. An stochastic reward model would be $p(boldsymbols', r|boldsymbols,boldsymbola)$ which requires an additional integral over reward $r$ (for example, equation (3.14) page 47)
Are my integral versions correct?
Yes. You are unfolding the recursive definition. An illustration would be the recursive definition for factorial:
$$f(n) = nf(n-1);f(0)=1$$
Which is unfolded as:
$$f(n) = n [(n-1) [(n-2)[...]]]$$
However, the difference is that the index in Bellman equation is going forward since current value depends on future values not the previous ones.
answered yesterday
EsmailianEsmailian
1,346113
$endgroup$
1
$begingroup$
Thank you for the confirmation. I didn’t realize the part with the stochastic reward.
$endgroup$
– MachineLearner
yesterday
add a comment |
Your Answer
StackExchange.ifUsing("editor", function ()
return StackExchange.using("mathjaxEditing", function ()
StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix)
StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["$", "$"], ["\\(","\\)"]]);
);
);
, "mathjax-editing");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "557"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
MachineLearner is a new contributor. Be nice, and check out our Code of Conduct.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f47425%2fhow-to-write-out-the-definition-of-the-value-function-for-continous-action-and-s%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
$begingroup$
this can be written as the integral, is this correct?
Yes. Your derivations imply that we have assumed a deterministic reward given current state-action $(boldsymbols,boldsymbola)$. An stochastic reward model would be $p(boldsymbols', r|boldsymbols,boldsymbola)$ which requires an additional integral over reward $r$ (for example, equation (3.14) page 47)
Are my integral versions correct?
Yes. You are unfolding the recursive definition. An illustration would be the recursive definition for factorial:
$$f(n) = nf(n-1);f(0)=1$$
Which is unfolded as:
$$f(n) = n [(n-1) [(n-2)[...]]]$$
However, the difference is that the index in Bellman equation is going forward since current value depends on future values not the previous ones.
answered yesterday
EsmailianEsmailian
1,346113
$endgroup$
1
$begingroup$
Thank you for the confirmation. I didn’t realize the part with the stochastic reward.
$endgroup$
– MachineLearner
yesterday
add a comment |
$begingroup$
this can be written as the integral, is this correct?
Yes. Your derivations imply that we have assumed a deterministic reward given current state-action $(boldsymbols,boldsymbola)$. An stochastic reward model would be $p(boldsymbols', r|boldsymbols,boldsymbola)$ which requires an additional integral over reward $r$ (for example, equation (3.14) page 47)
Are my integral versions correct?
Yes. You are unfolding the recursive definition. An illustration would be the recursive definition for factorial:
$$f(n) = nf(n-1);f(0)=1$$
Which is unfolded as:
$$f(n) = n [(n-1) [(n-2)[...]]]$$
However, the difference is that the index in Bellman equation is going forward since current value depends on future values not the previous ones.
answered yesterday
EsmailianEsmailian
1,346113
$endgroup$
1
$begingroup$
Thank you for the confirmation. I didn’t realize the part with the stochastic reward.
$endgroup$
– MachineLearner
yesterday
add a comment |
$begingroup$
this can be written as the integral, is this correct?
Yes. Your derivations imply that we have assumed a deterministic reward given current state-action $(boldsymbols,boldsymbola)$. An stochastic reward model would be $p(boldsymbols', r|boldsymbols,boldsymbola)$ which requires an additional integral over reward $r$ (for example, equation (3.14) page 47)
Are my integral versions correct?
Yes. You are unfolding the recursive definition. An illustration would be the recursive definition for factorial:
$$f(n) = nf(n-1);f(0)=1$$
Which is unfolded as:
$$f(n) = n [(n-1) [(n-2)[...]]]$$
However, the difference is that the index in Bellman equation is going forward since current value depends on future values not the previous ones.
answered yesterday
EsmailianEsmailian
1,346113
$endgroup$
this can be written as the integral, is this correct?
Yes. Your derivations imply that we have assumed a deterministic reward given current state-action $(boldsymbols,boldsymbola)$. An stochastic reward model would be $p(boldsymbols', r|boldsymbols,boldsymbola)$ which requires an additional integral over reward $r$ (for example, equation (3.14) page 47)
Are my integral versions correct?
Yes. You are unfolding the recursive definition. An illustration would be the recursive definition for factorial:
$$f(n) = nf(n-1);f(0)=1$$
Which is unfolded as:
$$f(n) = n [(n-1) [(n-2)[...]]]$$
However, the difference is that the index in Bellman equation is going forward since current value depends on future values not the previous ones.
answered yesterday
EsmailianEsmailian
1,346113
edited yesterday
answered yesterday
EsmailianEsmailian
1,346113
answered yesterday
EsmailianEsmailian
1,346113
answered yesterday
EsmailianEsmailian
1,346113
1,346113
1
$begingroup$
Thank you for the confirmation. I didn’t realize the part with the stochastic reward.
$endgroup$
– MachineLearner
yesterday
add a comment |
1
$begingroup$
Thank you for the confirmation. I didn’t realize the part with the stochastic reward.
$endgroup$
– MachineLearner
yesterday
1
1
$begingroup$
Thank you for the confirmation. I didn’t realize the part with the stochastic reward.
$endgroup$
– MachineLearner
yesterday
$begingroup$
Thank you for the confirmation. I didn’t realize the part with the stochastic reward.
$endgroup$
– MachineLearner
yesterday
add a comment |
MachineLearner is a new contributor. Be nice, and check out our Code of Conduct.
MachineLearner is a new contributor. Be nice, and check out our Code of Conduct.
MachineLearner is a new contributor. Be nice, and check out our Code of Conduct.
MachineLearner is a new contributor. Be nice, and check out our Code of Conduct.
Thanks for contributing an answer to Data Science Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
Use MathJax to format equations. MathJax reference.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f47425%2fhow-to-write-out-the-definition-of-the-value-function-for-continous-action-and-s%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
