review: gradient descent, epochs, validation in neural network trainingWhy mini batch size is better than one single “batch” with all training data?What is missing from the following Curriculum Learning implementation in a Deep Neural Net?Batching in Recurrent Neural Networks (RNNs) when there is only a single instance per time step?Neural Network: how to interpret this loss graph?Backpropagation with step or threshold activation functionDeep learning with Tensorflow: training with big data setsValidation loss keeps fluctuating about training lossFeed-forward neural network not training with Keras function generators deep_learning data_science machine_learning pythonValidation accuracy is always close to training accuracyHow is Stochastic Gradient Descent done in Faster RCNN?what exactly happens during each epoch in neural network training
How to deal with a cynical class?
If I can solve Sudoku can I solve TSP? If yes, how?
How to write cleanly even if my character uses expletive language?
All function values have been reset after restarting Mathematica
How could a scammer know the apps on my phone / iTunes account?
newcommand: Combine (optional) star and optional parameter
Ban on all campaign finance?
Are all passive ability checks floors for active ability checks?
Can anyone tell me why this program fails?
Rules about breaking the rules. How do I do it well?
How can I track script wich gives me "command not found" right after the login?
Have researchers managed to "reverse time"? If so, what does that mean for physics?
How to terminate ping <URL> &
Is it normal that my co-workers at a fitness company criticize my food choices?
Happy pi day, everyone!
Why is the BSI not using powers of two?
My adviser wants to be the first author
Min function accepting varying number of arguments in C++17
AG Cluster db upgrade by vendor
How to change two letters closest to a string and one letter immediately after a string using notepad++
Employee lack of ownership
Brexit - No Deal Rejection
Problems with making formula look great
PTIJ: Who should I vote for? (21st Knesset Edition)
review: gradient descent, epochs, validation in neural network training
Why mini batch size is better than one single “batch” with all training data?What is missing from the following Curriculum Learning implementation in a Deep Neural Net?Batching in Recurrent Neural Networks (RNNs) when there is only a single instance per time step?Neural Network: how to interpret this loss graph?Backpropagation with step or threshold activation functionDeep learning with Tensorflow: training with big data setsValidation loss keeps fluctuating about training lossFeed-forward neural network not training with Keras function generators deep_learning data_science machine_learning pythonValidation accuracy is always close to training accuracyHow is Stochastic Gradient Descent done in Faster RCNN?what exactly happens during each epoch in neural network training
$begingroup$
These days, training data aren't put in gradient descent all at once. Rather, they are put in batch after batch. Gradient descent is run once for each batch of training data. When all batches are traversed, 1 epoch is done. However, gradient descent hasn't minimized the loss function yet, the loss function still being on a slope. The loss function now and weights serve as initial values for the next epoch. Weights are initialized only at the beginning of the 1st batch of the 1st epoch.
During each epoch, the validation set is used to calculate the error (evaluated using trained weights) on the validation set. If the error is dropping, finish this epoch and go to the next epoch. Chances are, when the validation error is minimized, the current epoch isn't traversed yet, the remaining batches of training data are left unused.
If the above is all right, seems that the hyper parameters: number of ConvNet filters, size of ConvNet filters, number of layers won't be trained or tuned?
deep-learning cnn training hyperparameter-tuning
asked yesterday
feynmanfeynman
708
$endgroup$
add a comment |
$begingroup$
These days, training data aren't put in gradient descent all at once. Rather, they are put in batch after batch. Gradient descent is run once for each batch of training data. When all batches are traversed, 1 epoch is done. However, gradient descent hasn't minimized the loss function yet, the loss function still being on a slope. The loss function now and weights serve as initial values for the next epoch. Weights are initialized only at the beginning of the 1st batch of the 1st epoch.
During each epoch, the validation set is used to calculate the error (evaluated using trained weights) on the validation set. If the error is dropping, finish this epoch and go to the next epoch. Chances are, when the validation error is minimized, the current epoch isn't traversed yet, the remaining batches of training data are left unused.
If the above is all right, seems that the hyper parameters: number of ConvNet filters, size of ConvNet filters, number of layers won't be trained or tuned?
deep-learning cnn training hyperparameter-tuning
asked yesterday
feynmanfeynman
708
$endgroup$
add a comment |
$begingroup$
These days, training data aren't put in gradient descent all at once. Rather, they are put in batch after batch. Gradient descent is run once for each batch of training data. When all batches are traversed, 1 epoch is done. However, gradient descent hasn't minimized the loss function yet, the loss function still being on a slope. The loss function now and weights serve as initial values for the next epoch. Weights are initialized only at the beginning of the 1st batch of the 1st epoch.
During each epoch, the validation set is used to calculate the error (evaluated using trained weights) on the validation set. If the error is dropping, finish this epoch and go to the next epoch. Chances are, when the validation error is minimized, the current epoch isn't traversed yet, the remaining batches of training data are left unused.
If the above is all right, seems that the hyper parameters: number of ConvNet filters, size of ConvNet filters, number of layers won't be trained or tuned?
deep-learning cnn training hyperparameter-tuning
asked yesterday
feynmanfeynman
708
$endgroup$
These days, training data aren't put in gradient descent all at once. Rather, they are put in batch after batch. Gradient descent is run once for each batch of training data. When all batches are traversed, 1 epoch is done. However, gradient descent hasn't minimized the loss function yet, the loss function still being on a slope. The loss function now and weights serve as initial values for the next epoch. Weights are initialized only at the beginning of the 1st batch of the 1st epoch.
During each epoch, the validation set is used to calculate the error (evaluated using trained weights) on the validation set. If the error is dropping, finish this epoch and go to the next epoch. Chances are, when the validation error is minimized, the current epoch isn't traversed yet, the remaining batches of training data are left unused.
If the above is all right, seems that the hyper parameters: number of ConvNet filters, size of ConvNet filters, number of layers won't be trained or tuned?
deep-learning cnn training hyperparameter-tuning
deep-learning cnn training hyperparameter-tuning
asked yesterday
feynmanfeynman
708
asked yesterday
feynmanfeynman
708
asked yesterday
feynmanfeynman
708
asked yesterday
feynmanfeynman
708
asked yesterday
feynmanfeynman
708
708
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
$begingroup$
Chances are, when the validation error is minimized, the current epoch isn't traversed yet, the remaining batches of training data are left unused.
Typically, validation is done only at the end of the epoch. So no batch should remain unused. For example, this is the behavior of Keras https://keras.io/models/model/#fit
validation_freq: Only relevant if validation data is provided. Integer or list/tuple/set. If an integer, specifies how many training epochs to run before a new validation run is performed, e.g. validation_freq=2 runs validation every 2 epochs. If a list, tuple, or set, specifies the epochs on which to run validation, e.g. validation_freq=[1, 2, 10] runs validation at the end of the 1st, 2nd, and 10th epochs.
hyper parameters: number of ConvNet filters, size of ConvNet filters,
number of layers won't be trained or tuned?
Yes, such hyper parameters are set when model is defined. These are not modified during training.
However : Systems like Google AutoML are trying to change this. These systems try to learn model hyper parameters during training.
answered yesterday
Shamit VermaShamit Verma
82428
$endgroup$
$begingroup$
thx so much for correcting. but if validation is only done after a full epoch is finished, what's the point of sending training data batch by batch? when 1 batch goes thru gradient descent once, weights are updated once. if there's 10 batches in total, then gradient descent is gone thru 10 times in 1 epoch. if instead i put all the 10 batches together in, gradient descent is gone thru only once in this epoch. the result weight updates for these 2 scenarios r the same
$endgroup$
– feynman
yesterday
1
$begingroup$
1. It is not possible to train 10 batches on GPUs . Typically GPU will have 4 - 16 GB memory. So, it does not have enough memory to load all data + model parameters
$endgroup$
– Shamit Verma
yesterday
$begingroup$
2. The high througput may also translate to faster convergence depending on the variance in the dataset and the learning rate used.
$endgroup$
– Shamit Verma
yesterday
1
$begingroup$
3. For smaller data sets, model can be trained without batching.
$endgroup$
– Shamit Verma
yesterday
1
$begingroup$
datascience.stackexchange.com/questions/16807/…
$endgroup$
– Shamit Verma
yesterday
|
show 2 more comments
Your Answer
StackExchange.ifUsing("editor", function ()
return StackExchange.using("mathjaxEditing", function ()
StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix)
StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["$", "$"], ["\\(","\\)"]]);
);
);
, "mathjax-editing");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "557"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f47276%2freview-gradient-descent-epochs-validation-in-neural-network-training%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
$begingroup$
Chances are, when the validation error is minimized, the current epoch isn't traversed yet, the remaining batches of training data are left unused.
Typically, validation is done only at the end of the epoch. So no batch should remain unused. For example, this is the behavior of Keras https://keras.io/models/model/#fit
validation_freq: Only relevant if validation data is provided. Integer or list/tuple/set. If an integer, specifies how many training epochs to run before a new validation run is performed, e.g. validation_freq=2 runs validation every 2 epochs. If a list, tuple, or set, specifies the epochs on which to run validation, e.g. validation_freq=[1, 2, 10] runs validation at the end of the 1st, 2nd, and 10th epochs.
hyper parameters: number of ConvNet filters, size of ConvNet filters,
number of layers won't be trained or tuned?
Yes, such hyper parameters are set when model is defined. These are not modified during training.
However : Systems like Google AutoML are trying to change this. These systems try to learn model hyper parameters during training.
answered yesterday
Shamit VermaShamit Verma
82428
$endgroup$
$begingroup$
thx so much for correcting. but if validation is only done after a full epoch is finished, what's the point of sending training data batch by batch? when 1 batch goes thru gradient descent once, weights are updated once. if there's 10 batches in total, then gradient descent is gone thru 10 times in 1 epoch. if instead i put all the 10 batches together in, gradient descent is gone thru only once in this epoch. the result weight updates for these 2 scenarios r the same
$endgroup$
– feynman
yesterday
1
$begingroup$
1. It is not possible to train 10 batches on GPUs . Typically GPU will have 4 - 16 GB memory. So, it does not have enough memory to load all data + model parameters
$endgroup$
– Shamit Verma
yesterday
$begingroup$
2. The high througput may also translate to faster convergence depending on the variance in the dataset and the learning rate used.
$endgroup$
– Shamit Verma
yesterday
1
$begingroup$
3. For smaller data sets, model can be trained without batching.
$endgroup$
– Shamit Verma
yesterday
1
$begingroup$
datascience.stackexchange.com/questions/16807/…
$endgroup$
– Shamit Verma
yesterday
|
show 2 more comments
$begingroup$
Chances are, when the validation error is minimized, the current epoch isn't traversed yet, the remaining batches of training data are left unused.
Typically, validation is done only at the end of the epoch. So no batch should remain unused. For example, this is the behavior of Keras https://keras.io/models/model/#fit
validation_freq: Only relevant if validation data is provided. Integer or list/tuple/set. If an integer, specifies how many training epochs to run before a new validation run is performed, e.g. validation_freq=2 runs validation every 2 epochs. If a list, tuple, or set, specifies the epochs on which to run validation, e.g. validation_freq=[1, 2, 10] runs validation at the end of the 1st, 2nd, and 10th epochs.
hyper parameters: number of ConvNet filters, size of ConvNet filters,
number of layers won't be trained or tuned?
Yes, such hyper parameters are set when model is defined. These are not modified during training.
However : Systems like Google AutoML are trying to change this. These systems try to learn model hyper parameters during training.
answered yesterday
Shamit VermaShamit Verma
82428
$endgroup$
$begingroup$
thx so much for correcting. but if validation is only done after a full epoch is finished, what's the point of sending training data batch by batch? when 1 batch goes thru gradient descent once, weights are updated once. if there's 10 batches in total, then gradient descent is gone thru 10 times in 1 epoch. if instead i put all the 10 batches together in, gradient descent is gone thru only once in this epoch. the result weight updates for these 2 scenarios r the same
$endgroup$
– feynman
yesterday
1
$begingroup$
1. It is not possible to train 10 batches on GPUs . Typically GPU will have 4 - 16 GB memory. So, it does not have enough memory to load all data + model parameters
$endgroup$
– Shamit Verma
yesterday
$begingroup$
2. The high througput may also translate to faster convergence depending on the variance in the dataset and the learning rate used.
$endgroup$
– Shamit Verma
yesterday
1
$begingroup$
3. For smaller data sets, model can be trained without batching.
$endgroup$
– Shamit Verma
yesterday
1
$begingroup$
datascience.stackexchange.com/questions/16807/…
$endgroup$
– Shamit Verma
yesterday
|
show 2 more comments
$begingroup$
Chances are, when the validation error is minimized, the current epoch isn't traversed yet, the remaining batches of training data are left unused.
Typically, validation is done only at the end of the epoch. So no batch should remain unused. For example, this is the behavior of Keras https://keras.io/models/model/#fit
validation_freq: Only relevant if validation data is provided. Integer or list/tuple/set. If an integer, specifies how many training epochs to run before a new validation run is performed, e.g. validation_freq=2 runs validation every 2 epochs. If a list, tuple, or set, specifies the epochs on which to run validation, e.g. validation_freq=[1, 2, 10] runs validation at the end of the 1st, 2nd, and 10th epochs.
hyper parameters: number of ConvNet filters, size of ConvNet filters,
number of layers won't be trained or tuned?
Yes, such hyper parameters are set when model is defined. These are not modified during training.
However : Systems like Google AutoML are trying to change this. These systems try to learn model hyper parameters during training.
answered yesterday
Shamit VermaShamit Verma
82428
$endgroup$
Chances are, when the validation error is minimized, the current epoch isn't traversed yet, the remaining batches of training data are left unused.
Typically, validation is done only at the end of the epoch. So no batch should remain unused. For example, this is the behavior of Keras https://keras.io/models/model/#fit
validation_freq: Only relevant if validation data is provided. Integer or list/tuple/set. If an integer, specifies how many training epochs to run before a new validation run is performed, e.g. validation_freq=2 runs validation every 2 epochs. If a list, tuple, or set, specifies the epochs on which to run validation, e.g. validation_freq=[1, 2, 10] runs validation at the end of the 1st, 2nd, and 10th epochs.
hyper parameters: number of ConvNet filters, size of ConvNet filters,
number of layers won't be trained or tuned?
Yes, such hyper parameters are set when model is defined. These are not modified during training.
However : Systems like Google AutoML are trying to change this. These systems try to learn model hyper parameters during training.
answered yesterday
Shamit VermaShamit Verma
82428
answered yesterday
Shamit VermaShamit Verma
82428
answered yesterday
Shamit VermaShamit Verma
82428
answered yesterday
Shamit VermaShamit Verma
82428
82428
$begingroup$
thx so much for correcting. but if validation is only done after a full epoch is finished, what's the point of sending training data batch by batch? when 1 batch goes thru gradient descent once, weights are updated once. if there's 10 batches in total, then gradient descent is gone thru 10 times in 1 epoch. if instead i put all the 10 batches together in, gradient descent is gone thru only once in this epoch. the result weight updates for these 2 scenarios r the same
$endgroup$
– feynman
yesterday
1
$begingroup$
1. It is not possible to train 10 batches on GPUs . Typically GPU will have 4 - 16 GB memory. So, it does not have enough memory to load all data + model parameters
$endgroup$
– Shamit Verma
yesterday
$begingroup$
2. The high througput may also translate to faster convergence depending on the variance in the dataset and the learning rate used.
$endgroup$
– Shamit Verma
yesterday
1
$begingroup$
3. For smaller data sets, model can be trained without batching.
$endgroup$
– Shamit Verma
yesterday
1
$begingroup$
datascience.stackexchange.com/questions/16807/…
$endgroup$
– Shamit Verma
yesterday
|
show 2 more comments
$begingroup$
thx so much for correcting. but if validation is only done after a full epoch is finished, what's the point of sending training data batch by batch? when 1 batch goes thru gradient descent once, weights are updated once. if there's 10 batches in total, then gradient descent is gone thru 10 times in 1 epoch. if instead i put all the 10 batches together in, gradient descent is gone thru only once in this epoch. the result weight updates for these 2 scenarios r the same
$endgroup$
– feynman
yesterday
1
$begingroup$
1. It is not possible to train 10 batches on GPUs . Typically GPU will have 4 - 16 GB memory. So, it does not have enough memory to load all data + model parameters
$endgroup$
– Shamit Verma
yesterday
$begingroup$
2. The high througput may also translate to faster convergence depending on the variance in the dataset and the learning rate used.
$endgroup$
– Shamit Verma
yesterday
1
$begingroup$
3. For smaller data sets, model can be trained without batching.
$endgroup$
– Shamit Verma
yesterday
1
$begingroup$
datascience.stackexchange.com/questions/16807/…
$endgroup$
– Shamit Verma
yesterday
$begingroup$
thx so much for correcting. but if validation is only done after a full epoch is finished, what's the point of sending training data batch by batch? when 1 batch goes thru gradient descent once, weights are updated once. if there's 10 batches in total, then gradient descent is gone thru 10 times in 1 epoch. if instead i put all the 10 batches together in, gradient descent is gone thru only once in this epoch. the result weight updates for these 2 scenarios r the same
$endgroup$
– feynman
yesterday
$begingroup$
thx so much for correcting. but if validation is only done after a full epoch is finished, what's the point of sending training data batch by batch? when 1 batch goes thru gradient descent once, weights are updated once. if there's 10 batches in total, then gradient descent is gone thru 10 times in 1 epoch. if instead i put all the 10 batches together in, gradient descent is gone thru only once in this epoch. the result weight updates for these 2 scenarios r the same
$endgroup$
– feynman
yesterday
1
1
$begingroup$
1. It is not possible to train 10 batches on GPUs . Typically GPU will have 4 - 16 GB memory. So, it does not have enough memory to load all data + model parameters
$endgroup$
– Shamit Verma
yesterday
$begingroup$
1. It is not possible to train 10 batches on GPUs . Typically GPU will have 4 - 16 GB memory. So, it does not have enough memory to load all data + model parameters
$endgroup$
– Shamit Verma
yesterday
$begingroup$
2. The high througput may also translate to faster convergence depending on the variance in the dataset and the learning rate used.
$endgroup$
– Shamit Verma
yesterday
$begingroup$
2. The high througput may also translate to faster convergence depending on the variance in the dataset and the learning rate used.
$endgroup$
– Shamit Verma
yesterday
1
1
$begingroup$
3. For smaller data sets, model can be trained without batching.
$endgroup$
– Shamit Verma
yesterday
$begingroup$
3. For smaller data sets, model can be trained without batching.
$endgroup$
– Shamit Verma
yesterday
1
1
$begingroup$
datascience.stackexchange.com/questions/16807/…
$endgroup$
– Shamit Verma
yesterday
$begingroup$
datascience.stackexchange.com/questions/16807/…
$endgroup$
– Shamit Verma
yesterday
|
show 2 more comments
Thanks for contributing an answer to Data Science Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
Use MathJax to format equations. MathJax reference.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f47276%2freview-gradient-descent-epochs-validation-in-neural-network-training%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
