Trjtdtk

Where in the Bible does the greeting ("Dominus Vobiscum") used at Mass come from?

Mapping a list into a phase plot

Increase performance creating Mandelbrot set in python

Was Spock the First Vulcan in Starfleet?

Is there a problem with hiding "forgot password" until it's needed?

HashMap containsKey() returns false although hashCode() and equals() are true

How to prove that the query oracle is unitary?

Curses work by shouting - How to avoid collateral damage?

Is there an Impartial Brexit Deal comparison site?

How can I use the arrow sign in my bash prompt?

How could Frankenstein get the parts for his _second_ creature?

Cynical novel that describes an America ruled by the media, arms manufacturers, and ethnic figureheads

Is there any reason not to eat food that's been dropped on the surface of the moon?

What would be the benefits of having both a state and local currencies?

Finding all intervals that match predicate in vector

Everything Bob says is false. How does he get people to trust him?

Unattended/Unattended to?

Applicability of Single Responsibility Principle

when is out of tune ok?

Can I convert a rim brake wheel to a disc brake wheel?

Personal Teleportation as a Weapon

Displaying the order of the columns of a table

The baby cries all morning

What defines a dissertation?

Is stochastic gradient descent pseudo-stochastic?

Why do neural network researchers care about epochs?Repeated training examples in Gradient DescentConvergence Criteria for Stochastic Gradient DescentWhy do neural network researchers care about epochs?Parallel minibatch gradient descent algorithmsGradient Descent (GD) vs Stochastic Gradient Descent (SGD)How backpropagation through gradient descent represents the error after each forward passStochastic Gradient Descent, Mini-Batch and Batch Gradient DescentStochastic gradient descent Vs Mini-batch size 1Stochastic gradient descent vs mini-batch gradient descentSpecifics on weight update calculation in stochastic gradient descent

4

$begingroup$

I know that stochastic gradient descent randomly chooses 1 sample to update the weights. An epoch is defined as using all $N$ samples. So with SGD, for each epoch, we update the weights $N$ times.

My confusion is doesn't this make it so you have to go through all $N$ samples before you can see the same sample twice? Doesn't that effectively make it pseudo-random/stochastic? If it was entirely random, then there would be a possibility of seeing the same sample more than once before going through all $N$ samples.

machine-learning neural-networks gradient-descent sgd

share|cite|improve this question

edited Mar 20 at 15:50

Sycorax

42k12109207

asked Mar 20 at 15:14

IamanonIamanon

303

$endgroup$

add a comment | 

4

$begingroup$

I know that stochastic gradient descent randomly chooses 1 sample to update the weights. An epoch is defined as using all $N$ samples. So with SGD, for each epoch, we update the weights $N$ times.

My confusion is doesn't this make it so you have to go through all $N$ samples before you can see the same sample twice? Doesn't that effectively make it pseudo-random/stochastic? If it was entirely random, then there would be a possibility of seeing the same sample more than once before going through all $N$ samples.

machine-learning neural-networks gradient-descent sgd

share|cite|improve this question

edited Mar 20 at 15:50

Sycorax

42k12109207

asked Mar 20 at 15:14

IamanonIamanon

303

$endgroup$

add a comment | 

$begingroup$

I know that stochastic gradient descent randomly chooses 1 sample to update the weights. An epoch is defined as using all $N$ samples. So with SGD, for each epoch, we update the weights $N$ times.

My confusion is doesn't this make it so you have to go through all $N$ samples before you can see the same sample twice? Doesn't that effectively make it pseudo-random/stochastic? If it was entirely random, then there would be a possibility of seeing the same sample more than once before going through all $N$ samples.

machine-learning neural-networks gradient-descent sgd

share|cite|improve this question

edited Mar 20 at 15:50

Sycorax

42k12109207

asked Mar 20 at 15:14

IamanonIamanon

303

$endgroup$

share|cite|improve this question

edited Mar 20 at 15:50

Sycorax

42k12109207

asked Mar 20 at 15:14

IamanonIamanon

303

share|cite|improve this question

edited Mar 20 at 15:50

Sycorax

42k12109207

asked Mar 20 at 15:14

IamanonIamanon

303

edited Mar 20 at 15:50

Sycorax

42k12109207

edited Mar 20 at 15:50

Sycorax

42k12109207

asked Mar 20 at 15:14

IamanonIamanon

303

asked Mar 20 at 15:14

IamanonIamanon

303

1 Answer
1

active

oldest

votes

6

$begingroup$

Exhausting all $N$ samples before being able to repeat a sample means that the process is not independent. However, the process is still stochastic.

Consider a shuffled deck of cards. You look at the top card and see $mathsfAspadesuit$ (Ace of Spades), and set it aside. You'll never see another $mathsfAspadesuit$ in the whole deck. However, you don't know anything about the ordering of the remaining 51 cards, because the deck is shuffled. In this sense, the remainder of the deck still has a random order. The next card could be a $mathsf2colorredheartsuit$ or $mathsfJclubsuit$. You don't know for sure; all you do know is that the next card isn't the Ace of Spades, because you've put the only $mathsfAspadesuit$ face-up somewhere else.

In the scenario you outline, you're suggesting looking at the top card and then shuffling it into the deck again. This implies that the probability of seeing the $mathsfAspadesuit$ is independent of the previously-observed cards. Independence of events is an important attribute in probability theory, but it is not required to define a random process.

You might wonder why a person would want to construct mini-batches using the non-independent strategy. That question is answered here: Why do neural network researchers care about epochs?

share|cite|improve this answer

edited Mar 20 at 16:20

answered Mar 20 at 15:39

SycoraxSycorax

42k12109207

$endgroup$

add a comment | 

Your Answer

StackExchange.ifUsing("editor", function ()
return StackExchange.using("mathjaxEditing", function ()
StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix)
StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["$", "$"], ["\\(","\\)"]]);
);
);
, "mathjax-editing");

StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "65"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);

else
createEditor();

);

function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);



);

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);

Sign up using Google

Sign up using Facebook

Sign up using Email and Password

Post as a guest

Name

Email

Required, but never shown

StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstats.stackexchange.com%2fquestions%2f398540%2fis-stochastic-gradient-descent-pseudo-stochastic%23new-answer', 'question_page');

);

Post as a guest

Name

Email

Required, but never shown

6

$begingroup$

Exhausting all $N$ samples before being able to repeat a sample means that the process is not independent. However, the process is still stochastic.

Consider a shuffled deck of cards. You look at the top card and see $mathsfAspadesuit$ (Ace of Spades), and set it aside. You'll never see another $mathsfAspadesuit$ in the whole deck. However, you don't know anything about the ordering of the remaining 51 cards, because the deck is shuffled. In this sense, the remainder of the deck still has a random order. The next card could be a $mathsf2colorredheartsuit$ or $mathsfJclubsuit$. You don't know for sure; all you do know is that the next card isn't the Ace of Spades, because you've put the only $mathsfAspadesuit$ face-up somewhere else.

In the scenario you outline, you're suggesting looking at the top card and then shuffling it into the deck again. This implies that the probability of seeing the $mathsfAspadesuit$ is independent of the previously-observed cards. Independence of events is an important attribute in probability theory, but it is not required to define a random process.

You might wonder why a person would want to construct mini-batches using the non-independent strategy. That question is answered here: Why do neural network researchers care about epochs?

share|cite|improve this answer

edited Mar 20 at 16:20

answered Mar 20 at 15:39

SycoraxSycorax

42k12109207

$endgroup$

add a comment | 

6

$begingroup$

Exhausting all $N$ samples before being able to repeat a sample means that the process is not independent. However, the process is still stochastic.

Consider a shuffled deck of cards. You look at the top card and see $mathsfAspadesuit$ (Ace of Spades), and set it aside. You'll never see another $mathsfAspadesuit$ in the whole deck. However, you don't know anything about the ordering of the remaining 51 cards, because the deck is shuffled. In this sense, the remainder of the deck still has a random order. The next card could be a $mathsf2colorredheartsuit$ or $mathsfJclubsuit$. You don't know for sure; all you do know is that the next card isn't the Ace of Spades, because you've put the only $mathsfAspadesuit$ face-up somewhere else.

In the scenario you outline, you're suggesting looking at the top card and then shuffling it into the deck again. This implies that the probability of seeing the $mathsfAspadesuit$ is independent of the previously-observed cards. Independence of events is an important attribute in probability theory, but it is not required to define a random process.

You might wonder why a person would want to construct mini-batches using the non-independent strategy. That question is answered here: Why do neural network researchers care about epochs?

share|cite|improve this answer

edited Mar 20 at 16:20

answered Mar 20 at 15:39

SycoraxSycorax

42k12109207

$endgroup$

add a comment | 

$begingroup$

Exhausting all $N$ samples before being able to repeat a sample means that the process is not independent. However, the process is still stochastic.

Consider a shuffled deck of cards. You look at the top card and see $mathsfAspadesuit$ (Ace of Spades), and set it aside. You'll never see another $mathsfAspadesuit$ in the whole deck. However, you don't know anything about the ordering of the remaining 51 cards, because the deck is shuffled. In this sense, the remainder of the deck still has a random order. The next card could be a $mathsf2colorredheartsuit$ or $mathsfJclubsuit$. You don't know for sure; all you do know is that the next card isn't the Ace of Spades, because you've put the only $mathsfAspadesuit$ face-up somewhere else.

In the scenario you outline, you're suggesting looking at the top card and then shuffling it into the deck again. This implies that the probability of seeing the $mathsfAspadesuit$ is independent of the previously-observed cards. Independence of events is an important attribute in probability theory, but it is not required to define a random process.

You might wonder why a person would want to construct mini-batches using the non-independent strategy. That question is answered here: Why do neural network researchers care about epochs?

share|cite|improve this answer

edited Mar 20 at 16:20

answered Mar 20 at 15:39

SycoraxSycorax

42k12109207

$endgroup$

share|cite|improve this answer

edited Mar 20 at 16:20

answered Mar 20 at 15:39

SycoraxSycorax

42k12109207

answered Mar 20 at 15:39

SycoraxSycorax

42k12109207

answered Mar 20 at 15:39

SycoraxSycorax

42k12109207