What to set in steps_per_epoch in Keras' fit_generator?How to Create Shared Weights Layer in KerasHow to set batch_size, steps_per epoch and validation stepsKeras CNN image input and outputCustom Metrics with KerasKeras custom loss using multiple inputKeras intuition/guidelines for setting epochs and batch sizeBatch Size of Stateful LSTM in kerasEarly stopping and final Loss or weights of modelsValidation-split of Keras fit functionKeras Fit Generator Function

What does "Four-F." mean?

Can a wizard cast a spell during their first turn of combat if they initiated combat by releasing a readied spell?

How to generate binary array whose elements with values 1 are randomly drawn

Violin - Can double stops be played when the strings are not next to each other?

Is there a term for accumulated dirt on the outside of your hands and feet?

Should I be concerned about student access to a test bank?

Is there a hypothetical scenario that would make Earth uninhabitable for humans, but not for (the majority of) other animals?

Is it true that good novels will automatically sell themselves on Amazon (and so on) and there is no need for one to waste time promoting?

두음법칙 - When did North and South diverge in pronunciation of initial ㄹ?

PTIJ What is the inyan of the Konami code in Uncle Moishy's song?

If "dar" means "to give", what does "daros" mean?

Do I need to consider instance restrictions when showing a language is in P?

Pronounciation of the combination "st" in spanish accents

What (if any) is the reason to buy in small local stores?

What exactly term 'companion plants' means?

Maths symbols and unicode-math input inside siunitx commands

What can I do if I am asked to learn different programming languages very frequently?

Writing in a Christian voice

Brake pads destroying wheels

I got the following comment from a reputed math journal. What does it mean?

Probably overheated black color SMD pads

Using Past-Perfect interchangeably with the Past Continuous

Practical application of matrices and determinants

Is it insecure to send a password in a `curl` command?



What to set in steps_per_epoch in Keras' fit_generator?


How to Create Shared Weights Layer in KerasHow to set batch_size, steps_per epoch and validation stepsKeras CNN image input and outputCustom Metrics with KerasKeras custom loss using multiple inputKeras intuition/guidelines for setting epochs and batch sizeBatch Size of Stateful LSTM in kerasEarly stopping and final Loss or weights of modelsValidation-split of Keras fit functionKeras Fit Generator Function













1












$begingroup$


I am replicating in Keras the work of a paper where I know the values of epoch and batch_size. As the dataset is quite large I am using fit_generator, I want to know what to set in steps_per_epoch given epoch value and batch_size. Is there a standard way or something?










share|improve this question











$endgroup$
















    1












    $begingroup$


    I am replicating in Keras the work of a paper where I know the values of epoch and batch_size. As the dataset is quite large I am using fit_generator, I want to know what to set in steps_per_epoch given epoch value and batch_size. Is there a standard way or something?










    share|improve this question











    $endgroup$














      1












      1








      1





      $begingroup$


      I am replicating in Keras the work of a paper where I know the values of epoch and batch_size. As the dataset is quite large I am using fit_generator, I want to know what to set in steps_per_epoch given epoch value and batch_size. Is there a standard way or something?










      share|improve this question











      $endgroup$




      I am replicating in Keras the work of a paper where I know the values of epoch and batch_size. As the dataset is quite large I am using fit_generator, I want to know what to set in steps_per_epoch given epoch value and batch_size. Is there a standard way or something?







      keras epochs






      share|improve this question















      share|improve this question













      share|improve this question




      share|improve this question








      edited yesterday









      pcko1

      1,551317




      1,551317










      asked yesterday









      yamini goelyamini goel

      213




      213




















          2 Answers
          2






          active

          oldest

          votes


















          0












          $begingroup$

          I think it would be nice to have the following relation hold



          steps_per_epoch * batch_size = number_of_rows_in_train_data


          This will result in usage of all the train data for one epoch.
          Also, consider using fit() instead of fit_generator() if you need to have fast performance, but take into account that fit() might use more memory.






          share|improve this answer








          New contributor




          Karen Danielyan is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
          Check out our Code of Conduct.






          $endgroup$




















            0












            $begingroup$

            As mentioned in Keras' webpage about fit_generator():




            steps_per_epoch: Integer. Total number of steps (batches of samples)
            to yield from generator before declaring one epoch finished and
            starting the next epoch. It should typically be equal to
            ceil(num_samples / batch_size). Optional for Sequence: if unspecified,
            will use the len(generator) as a number of steps.




            You can set it equal to num_samples // batch_size, which is a typical choice.



            However, steps_per_epoch give you the chance to "trick" the generator when updating the learning rate using ReduceLROnPlateuau() callback, because this callback checks the drop of the loss once each epoch has finished. If the loss has stagnated for a patience number of consecutive epochs, the callback decreases the learning rate to "slow-cook" the network. If your dataset is huge, as it is usually the case when you need to use generators, you would probably like to decay the learning rate within a single epoch (since it includes a big number of data). This can be achieved by setting steps_per_epoch to a value that is less than num_samples // batch_size without affecting the overall number of training epochs of your model.



            Imagine this case as using mini-epochs within your normal epochs to change the learning rate because your loss has stagnated. I have found it very useful in my applications.






            share|improve this answer











            $endgroup$












            • $begingroup$
              I saw the documentation before posting the question, I couldn't understand what it means by num_samples. Does it mean total number of rows in x_train file?
              $endgroup$
              – yamini goel
              yesterday










            • $begingroup$
              it means the size of your dataset, which should be equal to the number of the rows in your file
              $endgroup$
              – pcko1
              yesterday










            • $begingroup$
              also, if I've answered your question sufficiently, please accept my answer
              $endgroup$
              – pcko1
              yesterday










            Your Answer





            StackExchange.ifUsing("editor", function ()
            return StackExchange.using("mathjaxEditing", function ()
            StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix)
            StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["$", "$"], ["\\(","\\)"]]);
            );
            );
            , "mathjax-editing");

            StackExchange.ready(function()
            var channelOptions =
            tags: "".split(" "),
            id: "557"
            ;
            initTagRenderer("".split(" "), "".split(" "), channelOptions);

            StackExchange.using("externalEditor", function()
            // Have to fire editor after snippets, if snippets enabled
            if (StackExchange.settings.snippets.snippetsEnabled)
            StackExchange.using("snippets", function()
            createEditor();
            );

            else
            createEditor();

            );

            function createEditor()
            StackExchange.prepareEditor(
            heartbeatType: 'answer',
            autoActivateHeartbeat: false,
            convertImagesToLinks: false,
            noModals: true,
            showLowRepImageUploadWarning: true,
            reputationToPostImages: null,
            bindNavPrevention: true,
            postfix: "",
            imageUploader:
            brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
            contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
            allowUrls: true
            ,
            onDemand: true,
            discardSelector: ".discard-answer"
            ,immediatelyShowMarkdownHelp:true
            );



            );













            draft saved

            draft discarded


















            StackExchange.ready(
            function ()
            StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f47405%2fwhat-to-set-in-steps-per-epoch-in-keras-fit-generator%23new-answer', 'question_page');

            );

            Post as a guest















            Required, but never shown

























            2 Answers
            2






            active

            oldest

            votes








            2 Answers
            2






            active

            oldest

            votes









            active

            oldest

            votes






            active

            oldest

            votes









            0












            $begingroup$

            I think it would be nice to have the following relation hold



            steps_per_epoch * batch_size = number_of_rows_in_train_data


            This will result in usage of all the train data for one epoch.
            Also, consider using fit() instead of fit_generator() if you need to have fast performance, but take into account that fit() might use more memory.






            share|improve this answer








            New contributor




            Karen Danielyan is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
            Check out our Code of Conduct.






            $endgroup$

















              0












              $begingroup$

              I think it would be nice to have the following relation hold



              steps_per_epoch * batch_size = number_of_rows_in_train_data


              This will result in usage of all the train data for one epoch.
              Also, consider using fit() instead of fit_generator() if you need to have fast performance, but take into account that fit() might use more memory.






              share|improve this answer








              New contributor




              Karen Danielyan is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
              Check out our Code of Conduct.






              $endgroup$















                0












                0








                0





                $begingroup$

                I think it would be nice to have the following relation hold



                steps_per_epoch * batch_size = number_of_rows_in_train_data


                This will result in usage of all the train data for one epoch.
                Also, consider using fit() instead of fit_generator() if you need to have fast performance, but take into account that fit() might use more memory.






                share|improve this answer








                New contributor




                Karen Danielyan is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
                Check out our Code of Conduct.






                $endgroup$



                I think it would be nice to have the following relation hold



                steps_per_epoch * batch_size = number_of_rows_in_train_data


                This will result in usage of all the train data for one epoch.
                Also, consider using fit() instead of fit_generator() if you need to have fast performance, but take into account that fit() might use more memory.







                share|improve this answer








                New contributor




                Karen Danielyan is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
                Check out our Code of Conduct.









                share|improve this answer



                share|improve this answer






                New contributor




                Karen Danielyan is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
                Check out our Code of Conduct.









                answered yesterday









                Karen DanielyanKaren Danielyan

                16




                16




                New contributor




                Karen Danielyan is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
                Check out our Code of Conduct.





                New contributor





                Karen Danielyan is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
                Check out our Code of Conduct.






                Karen Danielyan is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
                Check out our Code of Conduct.





















                    0












                    $begingroup$

                    As mentioned in Keras' webpage about fit_generator():




                    steps_per_epoch: Integer. Total number of steps (batches of samples)
                    to yield from generator before declaring one epoch finished and
                    starting the next epoch. It should typically be equal to
                    ceil(num_samples / batch_size). Optional for Sequence: if unspecified,
                    will use the len(generator) as a number of steps.




                    You can set it equal to num_samples // batch_size, which is a typical choice.



                    However, steps_per_epoch give you the chance to "trick" the generator when updating the learning rate using ReduceLROnPlateuau() callback, because this callback checks the drop of the loss once each epoch has finished. If the loss has stagnated for a patience number of consecutive epochs, the callback decreases the learning rate to "slow-cook" the network. If your dataset is huge, as it is usually the case when you need to use generators, you would probably like to decay the learning rate within a single epoch (since it includes a big number of data). This can be achieved by setting steps_per_epoch to a value that is less than num_samples // batch_size without affecting the overall number of training epochs of your model.



                    Imagine this case as using mini-epochs within your normal epochs to change the learning rate because your loss has stagnated. I have found it very useful in my applications.






                    share|improve this answer











                    $endgroup$












                    • $begingroup$
                      I saw the documentation before posting the question, I couldn't understand what it means by num_samples. Does it mean total number of rows in x_train file?
                      $endgroup$
                      – yamini goel
                      yesterday










                    • $begingroup$
                      it means the size of your dataset, which should be equal to the number of the rows in your file
                      $endgroup$
                      – pcko1
                      yesterday










                    • $begingroup$
                      also, if I've answered your question sufficiently, please accept my answer
                      $endgroup$
                      – pcko1
                      yesterday















                    0












                    $begingroup$

                    As mentioned in Keras' webpage about fit_generator():




                    steps_per_epoch: Integer. Total number of steps (batches of samples)
                    to yield from generator before declaring one epoch finished and
                    starting the next epoch. It should typically be equal to
                    ceil(num_samples / batch_size). Optional for Sequence: if unspecified,
                    will use the len(generator) as a number of steps.




                    You can set it equal to num_samples // batch_size, which is a typical choice.



                    However, steps_per_epoch give you the chance to "trick" the generator when updating the learning rate using ReduceLROnPlateuau() callback, because this callback checks the drop of the loss once each epoch has finished. If the loss has stagnated for a patience number of consecutive epochs, the callback decreases the learning rate to "slow-cook" the network. If your dataset is huge, as it is usually the case when you need to use generators, you would probably like to decay the learning rate within a single epoch (since it includes a big number of data). This can be achieved by setting steps_per_epoch to a value that is less than num_samples // batch_size without affecting the overall number of training epochs of your model.



                    Imagine this case as using mini-epochs within your normal epochs to change the learning rate because your loss has stagnated. I have found it very useful in my applications.






                    share|improve this answer











                    $endgroup$












                    • $begingroup$
                      I saw the documentation before posting the question, I couldn't understand what it means by num_samples. Does it mean total number of rows in x_train file?
                      $endgroup$
                      – yamini goel
                      yesterday










                    • $begingroup$
                      it means the size of your dataset, which should be equal to the number of the rows in your file
                      $endgroup$
                      – pcko1
                      yesterday










                    • $begingroup$
                      also, if I've answered your question sufficiently, please accept my answer
                      $endgroup$
                      – pcko1
                      yesterday













                    0












                    0








                    0





                    $begingroup$

                    As mentioned in Keras' webpage about fit_generator():




                    steps_per_epoch: Integer. Total number of steps (batches of samples)
                    to yield from generator before declaring one epoch finished and
                    starting the next epoch. It should typically be equal to
                    ceil(num_samples / batch_size). Optional for Sequence: if unspecified,
                    will use the len(generator) as a number of steps.




                    You can set it equal to num_samples // batch_size, which is a typical choice.



                    However, steps_per_epoch give you the chance to "trick" the generator when updating the learning rate using ReduceLROnPlateuau() callback, because this callback checks the drop of the loss once each epoch has finished. If the loss has stagnated for a patience number of consecutive epochs, the callback decreases the learning rate to "slow-cook" the network. If your dataset is huge, as it is usually the case when you need to use generators, you would probably like to decay the learning rate within a single epoch (since it includes a big number of data). This can be achieved by setting steps_per_epoch to a value that is less than num_samples // batch_size without affecting the overall number of training epochs of your model.



                    Imagine this case as using mini-epochs within your normal epochs to change the learning rate because your loss has stagnated. I have found it very useful in my applications.






                    share|improve this answer











                    $endgroup$



                    As mentioned in Keras' webpage about fit_generator():




                    steps_per_epoch: Integer. Total number of steps (batches of samples)
                    to yield from generator before declaring one epoch finished and
                    starting the next epoch. It should typically be equal to
                    ceil(num_samples / batch_size). Optional for Sequence: if unspecified,
                    will use the len(generator) as a number of steps.




                    You can set it equal to num_samples // batch_size, which is a typical choice.



                    However, steps_per_epoch give you the chance to "trick" the generator when updating the learning rate using ReduceLROnPlateuau() callback, because this callback checks the drop of the loss once each epoch has finished. If the loss has stagnated for a patience number of consecutive epochs, the callback decreases the learning rate to "slow-cook" the network. If your dataset is huge, as it is usually the case when you need to use generators, you would probably like to decay the learning rate within a single epoch (since it includes a big number of data). This can be achieved by setting steps_per_epoch to a value that is less than num_samples // batch_size without affecting the overall number of training epochs of your model.



                    Imagine this case as using mini-epochs within your normal epochs to change the learning rate because your loss has stagnated. I have found it very useful in my applications.







                    share|improve this answer














                    share|improve this answer



                    share|improve this answer








                    edited yesterday

























                    answered yesterday









                    pcko1pcko1

                    1,551317




                    1,551317











                    • $begingroup$
                      I saw the documentation before posting the question, I couldn't understand what it means by num_samples. Does it mean total number of rows in x_train file?
                      $endgroup$
                      – yamini goel
                      yesterday










                    • $begingroup$
                      it means the size of your dataset, which should be equal to the number of the rows in your file
                      $endgroup$
                      – pcko1
                      yesterday










                    • $begingroup$
                      also, if I've answered your question sufficiently, please accept my answer
                      $endgroup$
                      – pcko1
                      yesterday
















                    • $begingroup$
                      I saw the documentation before posting the question, I couldn't understand what it means by num_samples. Does it mean total number of rows in x_train file?
                      $endgroup$
                      – yamini goel
                      yesterday










                    • $begingroup$
                      it means the size of your dataset, which should be equal to the number of the rows in your file
                      $endgroup$
                      – pcko1
                      yesterday










                    • $begingroup$
                      also, if I've answered your question sufficiently, please accept my answer
                      $endgroup$
                      – pcko1
                      yesterday















                    $begingroup$
                    I saw the documentation before posting the question, I couldn't understand what it means by num_samples. Does it mean total number of rows in x_train file?
                    $endgroup$
                    – yamini goel
                    yesterday




                    $begingroup$
                    I saw the documentation before posting the question, I couldn't understand what it means by num_samples. Does it mean total number of rows in x_train file?
                    $endgroup$
                    – yamini goel
                    yesterday












                    $begingroup$
                    it means the size of your dataset, which should be equal to the number of the rows in your file
                    $endgroup$
                    – pcko1
                    yesterday




                    $begingroup$
                    it means the size of your dataset, which should be equal to the number of the rows in your file
                    $endgroup$
                    – pcko1
                    yesterday












                    $begingroup$
                    also, if I've answered your question sufficiently, please accept my answer
                    $endgroup$
                    – pcko1
                    yesterday




                    $begingroup$
                    also, if I've answered your question sufficiently, please accept my answer
                    $endgroup$
                    – pcko1
                    yesterday

















                    draft saved

                    draft discarded
















































                    Thanks for contributing an answer to Data Science Stack Exchange!


                    • Please be sure to answer the question. Provide details and share your research!

                    But avoid


                    • Asking for help, clarification, or responding to other answers.

                    • Making statements based on opinion; back them up with references or personal experience.

                    Use MathJax to format equations. MathJax reference.


                    To learn more, see our tips on writing great answers.




                    draft saved


                    draft discarded














                    StackExchange.ready(
                    function ()
                    StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f47405%2fwhat-to-set-in-steps-per-epoch-in-keras-fit-generator%23new-answer', 'question_page');

                    );

                    Post as a guest















                    Required, but never shown





















































                    Required, but never shown














                    Required, but never shown












                    Required, but never shown







                    Required, but never shown

































                    Required, but never shown














                    Required, but never shown












                    Required, but never shown







                    Required, but never shown







                    Popular posts from this blog

                    Adding axes to figuresAdding axes labels to LaTeX figuresLaTeX equivalent of ConTeXt buffersRotate a node but not its content: the case of the ellipse decorationHow to define the default vertical distance between nodes?TikZ scaling graphic and adjust node position and keep font sizeNumerical conditional within tikz keys?adding axes to shapesAlign axes across subfiguresAdding figures with a certain orderLine up nested tikz enviroments or how to get rid of themAdding axes labels to LaTeX figures

                    Tähtien Talli Jäsenet | Lähteet | NavigointivalikkoSuomen Hippos – Tähtien Talli

                    Do these cracks on my tires look bad? The Next CEO of Stack OverflowDry rot tire should I replace?Having to replace tiresFishtailed so easily? Bad tires? ABS?Filling the tires with something other than air, to avoid puncture hassles?Used Michelin tires safe to install?Do these tyre cracks necessitate replacement?Rumbling noise: tires or mechanicalIs it possible to fix noisy feathered tires?Are bad winter tires still better than summer tires in winter?Torque converter failure - Related to replacing only 2 tires?Why use snow tires on all 4 wheels on 2-wheel-drive cars?