What to set in steps_per_epoch in Keras' fit_generator?How to Create Shared Weights Layer in KerasHow to set batch_size, steps_per epoch and validation stepsKeras CNN image input and outputCustom Metrics with KerasKeras custom loss using multiple inputKeras intuition/guidelines for setting epochs and batch sizeBatch Size of Stateful LSTM in kerasEarly stopping and final Loss or weights of modelsValidation-split of Keras fit functionKeras Fit Generator Function
What does "Four-F." mean?
Can a wizard cast a spell during their first turn of combat if they initiated combat by releasing a readied spell?
How to generate binary array whose elements with values 1 are randomly drawn
Violin - Can double stops be played when the strings are not next to each other?
Is there a term for accumulated dirt on the outside of your hands and feet?
Should I be concerned about student access to a test bank?
Is there a hypothetical scenario that would make Earth uninhabitable for humans, but not for (the majority of) other animals?
Is it true that good novels will automatically sell themselves on Amazon (and so on) and there is no need for one to waste time promoting?
두음법칙 - When did North and South diverge in pronunciation of initial ㄹ?
PTIJ What is the inyan of the Konami code in Uncle Moishy's song?
If "dar" means "to give", what does "daros" mean?
Do I need to consider instance restrictions when showing a language is in P?
Pronounciation of the combination "st" in spanish accents
What (if any) is the reason to buy in small local stores?
What exactly term 'companion plants' means?
Maths symbols and unicode-math input inside siunitx commands
What can I do if I am asked to learn different programming languages very frequently?
Writing in a Christian voice
Brake pads destroying wheels
I got the following comment from a reputed math journal. What does it mean?
Probably overheated black color SMD pads
Using Past-Perfect interchangeably with the Past Continuous
Practical application of matrices and determinants
Is it insecure to send a password in a `curl` command?
What to set in steps_per_epoch in Keras' fit_generator?
How to Create Shared Weights Layer in KerasHow to set batch_size, steps_per epoch and validation stepsKeras CNN image input and outputCustom Metrics with KerasKeras custom loss using multiple inputKeras intuition/guidelines for setting epochs and batch sizeBatch Size of Stateful LSTM in kerasEarly stopping and final Loss or weights of modelsValidation-split of Keras fit functionKeras Fit Generator Function
$begingroup$
I am replicating in Keras the work of a paper where I know the values of epoch and batch_size. As the dataset is quite large I am using fit_generator, I want to know what to set in steps_per_epoch given epoch value and batch_size. Is there a standard way or something?
keras epochs
edited yesterday
pcko1
1,551317
asked yesterday
yamini goelyamini goel
213
$endgroup$
add a comment |
$begingroup$
I am replicating in Keras the work of a paper where I know the values of epoch and batch_size. As the dataset is quite large I am using fit_generator, I want to know what to set in steps_per_epoch given epoch value and batch_size. Is there a standard way or something?
keras epochs
edited yesterday
pcko1
1,551317
asked yesterday
yamini goelyamini goel
213
$endgroup$
add a comment |
$begingroup$
I am replicating in Keras the work of a paper where I know the values of epoch and batch_size. As the dataset is quite large I am using fit_generator, I want to know what to set in steps_per_epoch given epoch value and batch_size. Is there a standard way or something?
keras epochs
edited yesterday
pcko1
1,551317
asked yesterday
yamini goelyamini goel
213
$endgroup$
I am replicating in Keras the work of a paper where I know the values of epoch and batch_size. As the dataset is quite large I am using fit_generator, I want to know what to set in steps_per_epoch given epoch value and batch_size. Is there a standard way or something?
keras epochs
keras epochs
edited yesterday
pcko1
1,551317
asked yesterday
yamini goelyamini goel
213
edited yesterday
pcko1
1,551317
asked yesterday
yamini goelyamini goel
213
edited yesterday
pcko1
1,551317
edited yesterday
pcko1
1,551317
edited yesterday
pcko1
1,551317
1,551317
asked yesterday
yamini goelyamini goel
213
asked yesterday
yamini goelyamini goel
213
asked yesterday
yamini goelyamini goel
213
213
add a comment |
add a comment |
2 Answers
2
active
oldest
votes
$begingroup$
I think it would be nice to have the following relation hold
steps_per_epoch * batch_size = number_of_rows_in_train_data
This will result in usage of all the train data for one epoch.
Also, consider using fit() instead of fit_generator() if you need to have fast performance, but take into account that fit() might use more memory.
answered yesterday
Karen DanielyanKaren Danielyan
16
New contributor
Karen Danielyan is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
$endgroup$
add a comment |
$begingroup$
As mentioned in Keras' webpage about fit_generator():
steps_per_epoch: Integer. Total number of steps (batches of samples)
to yield from generator before declaring one epoch finished and
starting the next epoch. It should typically be equal to
ceil(num_samples / batch_size). Optional for Sequence: if unspecified,
will use the len(generator) as a number of steps.
You can set it equal to num_samples // batch_size, which is a typical choice.
However, steps_per_epoch give you the chance to "trick" the generator when updating the learning rate using ReduceLROnPlateuau() callback, because this callback checks the drop of the loss once each epoch has finished. If the loss has stagnated for a patience number of consecutive epochs, the callback decreases the learning rate to "slow-cook" the network. If your dataset is huge, as it is usually the case when you need to use generators, you would probably like to decay the learning rate within a single epoch (since it includes a big number of data). This can be achieved by setting steps_per_epoch to a value that is less than num_samples // batch_size without affecting the overall number of training epochs of your model.
Imagine this case as using mini-epochs within your normal epochs to change the learning rate because your loss has stagnated. I have found it very useful in my applications.
answered yesterday
pcko1pcko1
1,551317
$endgroup$
$begingroup$
I saw the documentation before posting the question, I couldn't understand what it means bynum_samples. Does it mean total number of rows inx_trainfile?
$endgroup$
– yamini goel
yesterday
$begingroup$
it means the size of your dataset, which should be equal to the number of the rows in your file
$endgroup$
– pcko1
yesterday
$begingroup$
also, if I've answered your question sufficiently, please accept my answer
$endgroup$
– pcko1
yesterday
add a comment |
Your Answer
StackExchange.ifUsing("editor", function ()
return StackExchange.using("mathjaxEditing", function ()
StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix)
StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["$", "$"], ["\\(","\\)"]]);
);
);
, "mathjax-editing");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "557"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f47405%2fwhat-to-set-in-steps-per-epoch-in-keras-fit-generator%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
2 Answers
2
active
oldest
votes
2 Answers
2
active
oldest
votes
active
oldest
votes
active
oldest
votes
$begingroup$
I think it would be nice to have the following relation hold
steps_per_epoch * batch_size = number_of_rows_in_train_data
This will result in usage of all the train data for one epoch.
Also, consider using fit() instead of fit_generator() if you need to have fast performance, but take into account that fit() might use more memory.
answered yesterday
Karen DanielyanKaren Danielyan
16
New contributor
Karen Danielyan is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
$endgroup$
add a comment |
$begingroup$
I think it would be nice to have the following relation hold
steps_per_epoch * batch_size = number_of_rows_in_train_data
This will result in usage of all the train data for one epoch.
Also, consider using fit() instead of fit_generator() if you need to have fast performance, but take into account that fit() might use more memory.
answered yesterday
Karen DanielyanKaren Danielyan
16
New contributor
Karen Danielyan is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
$endgroup$
add a comment |
$begingroup$
I think it would be nice to have the following relation hold
steps_per_epoch * batch_size = number_of_rows_in_train_data
This will result in usage of all the train data for one epoch.
Also, consider using fit() instead of fit_generator() if you need to have fast performance, but take into account that fit() might use more memory.
answered yesterday
Karen DanielyanKaren Danielyan
16
New contributor
Karen Danielyan is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
$endgroup$
I think it would be nice to have the following relation hold
steps_per_epoch * batch_size = number_of_rows_in_train_data
This will result in usage of all the train data for one epoch.
Also, consider using fit() instead of fit_generator() if you need to have fast performance, but take into account that fit() might use more memory.
answered yesterday
Karen DanielyanKaren Danielyan
16
New contributor
Karen Danielyan is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
answered yesterday
Karen DanielyanKaren Danielyan
16
New contributor
Karen Danielyan is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
answered yesterday
Karen DanielyanKaren Danielyan
16
answered yesterday
Karen DanielyanKaren Danielyan
16
16
New contributor
Karen Danielyan is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
New contributor
Karen Danielyan is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
Karen Danielyan is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
add a comment |
add a comment |
$begingroup$
As mentioned in Keras' webpage about fit_generator():
steps_per_epoch: Integer. Total number of steps (batches of samples)
to yield from generator before declaring one epoch finished and
starting the next epoch. It should typically be equal to
ceil(num_samples / batch_size). Optional for Sequence: if unspecified,
will use the len(generator) as a number of steps.
You can set it equal to num_samples // batch_size, which is a typical choice.
However, steps_per_epoch give you the chance to "trick" the generator when updating the learning rate using ReduceLROnPlateuau() callback, because this callback checks the drop of the loss once each epoch has finished. If the loss has stagnated for a patience number of consecutive epochs, the callback decreases the learning rate to "slow-cook" the network. If your dataset is huge, as it is usually the case when you need to use generators, you would probably like to decay the learning rate within a single epoch (since it includes a big number of data). This can be achieved by setting steps_per_epoch to a value that is less than num_samples // batch_size without affecting the overall number of training epochs of your model.
Imagine this case as using mini-epochs within your normal epochs to change the learning rate because your loss has stagnated. I have found it very useful in my applications.
answered yesterday
pcko1pcko1
1,551317
$endgroup$
$begingroup$
I saw the documentation before posting the question, I couldn't understand what it means bynum_samples. Does it mean total number of rows inx_trainfile?
$endgroup$
– yamini goel
yesterday
$begingroup$
it means the size of your dataset, which should be equal to the number of the rows in your file
$endgroup$
– pcko1
yesterday
$begingroup$
also, if I've answered your question sufficiently, please accept my answer
$endgroup$
– pcko1
yesterday
add a comment |
$begingroup$
As mentioned in Keras' webpage about fit_generator():
steps_per_epoch: Integer. Total number of steps (batches of samples)
to yield from generator before declaring one epoch finished and
starting the next epoch. It should typically be equal to
ceil(num_samples / batch_size). Optional for Sequence: if unspecified,
will use the len(generator) as a number of steps.
You can set it equal to num_samples // batch_size, which is a typical choice.
However, steps_per_epoch give you the chance to "trick" the generator when updating the learning rate using ReduceLROnPlateuau() callback, because this callback checks the drop of the loss once each epoch has finished. If the loss has stagnated for a patience number of consecutive epochs, the callback decreases the learning rate to "slow-cook" the network. If your dataset is huge, as it is usually the case when you need to use generators, you would probably like to decay the learning rate within a single epoch (since it includes a big number of data). This can be achieved by setting steps_per_epoch to a value that is less than num_samples // batch_size without affecting the overall number of training epochs of your model.
Imagine this case as using mini-epochs within your normal epochs to change the learning rate because your loss has stagnated. I have found it very useful in my applications.
answered yesterday
pcko1pcko1
1,551317
$endgroup$
$begingroup$
I saw the documentation before posting the question, I couldn't understand what it means bynum_samples. Does it mean total number of rows inx_trainfile?
$endgroup$
– yamini goel
yesterday
$begingroup$
it means the size of your dataset, which should be equal to the number of the rows in your file
$endgroup$
– pcko1
yesterday
$begingroup$
also, if I've answered your question sufficiently, please accept my answer
$endgroup$
– pcko1
yesterday
add a comment |
$begingroup$
As mentioned in Keras' webpage about fit_generator():
steps_per_epoch: Integer. Total number of steps (batches of samples)
to yield from generator before declaring one epoch finished and
starting the next epoch. It should typically be equal to
ceil(num_samples / batch_size). Optional for Sequence: if unspecified,
will use the len(generator) as a number of steps.
You can set it equal to num_samples // batch_size, which is a typical choice.
However, steps_per_epoch give you the chance to "trick" the generator when updating the learning rate using ReduceLROnPlateuau() callback, because this callback checks the drop of the loss once each epoch has finished. If the loss has stagnated for a patience number of consecutive epochs, the callback decreases the learning rate to "slow-cook" the network. If your dataset is huge, as it is usually the case when you need to use generators, you would probably like to decay the learning rate within a single epoch (since it includes a big number of data). This can be achieved by setting steps_per_epoch to a value that is less than num_samples // batch_size without affecting the overall number of training epochs of your model.
Imagine this case as using mini-epochs within your normal epochs to change the learning rate because your loss has stagnated. I have found it very useful in my applications.
answered yesterday
pcko1pcko1
1,551317
$endgroup$
As mentioned in Keras' webpage about fit_generator():
steps_per_epoch: Integer. Total number of steps (batches of samples)
to yield from generator before declaring one epoch finished and
starting the next epoch. It should typically be equal to
ceil(num_samples / batch_size). Optional for Sequence: if unspecified,
will use the len(generator) as a number of steps.
You can set it equal to num_samples // batch_size, which is a typical choice.
However, steps_per_epoch give you the chance to "trick" the generator when updating the learning rate using ReduceLROnPlateuau() callback, because this callback checks the drop of the loss once each epoch has finished. If the loss has stagnated for a patience number of consecutive epochs, the callback decreases the learning rate to "slow-cook" the network. If your dataset is huge, as it is usually the case when you need to use generators, you would probably like to decay the learning rate within a single epoch (since it includes a big number of data). This can be achieved by setting steps_per_epoch to a value that is less than num_samples // batch_size without affecting the overall number of training epochs of your model.
Imagine this case as using mini-epochs within your normal epochs to change the learning rate because your loss has stagnated. I have found it very useful in my applications.
answered yesterday
pcko1pcko1
1,551317
edited yesterday
answered yesterday
pcko1pcko1
1,551317
answered yesterday
pcko1pcko1
1,551317
answered yesterday
pcko1pcko1
1,551317
1,551317
$begingroup$
I saw the documentation before posting the question, I couldn't understand what it means bynum_samples. Does it mean total number of rows inx_trainfile?
$endgroup$
– yamini goel
yesterday
$begingroup$
it means the size of your dataset, which should be equal to the number of the rows in your file
$endgroup$
– pcko1
yesterday
$begingroup$
also, if I've answered your question sufficiently, please accept my answer
$endgroup$
– pcko1
yesterday
add a comment |
$begingroup$
I saw the documentation before posting the question, I couldn't understand what it means bynum_samples. Does it mean total number of rows inx_trainfile?
$endgroup$
– yamini goel
yesterday
$begingroup$
it means the size of your dataset, which should be equal to the number of the rows in your file
$endgroup$
– pcko1
yesterday
$begingroup$
also, if I've answered your question sufficiently, please accept my answer
$endgroup$
– pcko1
yesterday
$begingroup$
I saw the documentation before posting the question, I couldn't understand what it means by
num_samples. Does it mean total number of rows in x_train file?$endgroup$
– yamini goel
yesterday
$begingroup$
I saw the documentation before posting the question, I couldn't understand what it means by
num_samples. Does it mean total number of rows in x_train file?$endgroup$
– yamini goel
yesterday
$begingroup$
it means the size of your dataset, which should be equal to the number of the rows in your file
$endgroup$
– pcko1
yesterday
$begingroup$
it means the size of your dataset, which should be equal to the number of the rows in your file
$endgroup$
– pcko1
yesterday
$begingroup$
also, if I've answered your question sufficiently, please accept my answer
$endgroup$
– pcko1
yesterday
$begingroup$
also, if I've answered your question sufficiently, please accept my answer
$endgroup$
– pcko1
yesterday
add a comment |
Thanks for contributing an answer to Data Science Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
Use MathJax to format equations. MathJax reference.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f47405%2fwhat-to-set-in-steps-per-epoch-in-keras-fit-generator%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
