What causes the network validation loss to always be lower than train loss? The 2019 Stack Overflow Developer Survey Results Are InLSTM Validation MSE always lower than Train MSEHow to create a multiple layer perceptron with layers of specific sizes in keras?Simple prediction with KerasValueError: Error when checking target: expected dense_2 to have shape (1,) but got array with shape (0,)How to set input for proper fit with lstm?Understanding LSTM behaviour: Validation loss smaller than training loss throughout training for regression problemNeural Network Prediction regression task, output is a multiple factor of input with same peaksValidation loss is lower than the training lossIN CIFAR 10 DATASETHow to recognise when to stop training based on Overfitting/Underfitting?
Does a dangling wire really electrocute me if I'm standing in water?
What spell level should this homebrew After-Image spell be?
What tool would a Roman-age civilization have to grind silver and other metals into dust?
Realistic Alternatives to Dust: What Else Could Feed a Plankton Bloom?
Is it true that “A.D.” is traditionally placed before the year number?
What do hard-Brexiteers want with respect to the Irish border?
What is the use of option -o in the useradd command?
Is an up-to-date browser secure on an out-of-date OS?
Is domain driven design an anti-SQL pattern?
Is "plugging out" electronic devices an American expression?
Manuscript was "unsubmitted" because the manuscript was deposited in Arxiv Preprints
Should I use my personal e-mail address, or my workplace one, when registering to external websites for work purposes?
Unbreakable Formation vs. Cry of the Carnarium
Are there any other methods to apply to solving simultaneous equations?
How come people say “Would of”?
Potential by Assembling Charges
What is the motivation for a law requiring 2 parties to consent for recording a conversation
Rank groups within a grouped sequence of TRUE/FALSE and NA
aging parents with no investments
How do you say "canon" as in "official for a story universe"?
Why is my p-value correlated to difference between means in two sample tests?
Why do some words that are not inflected have an umlaut?
The difference between dialogue marks
How was Skylab's orbit inclination chosen?
What causes the network validation loss to always be lower than train loss?
The 2019 Stack Overflow Developer Survey Results Are InLSTM Validation MSE always lower than Train MSEHow to create a multiple layer perceptron with layers of specific sizes in keras?Simple prediction with KerasValueError: Error when checking target: expected dense_2 to have shape (1,) but got array with shape (0,)How to set input for proper fit with lstm?Understanding LSTM behaviour: Validation loss smaller than training loss throughout training for regression problemNeural Network Prediction regression task, output is a multiple factor of input with same peaksValidation loss is lower than the training lossIN CIFAR 10 DATASETHow to recognise when to stop training based on Overfitting/Underfitting?
$begingroup$
I am using a LSTM model to predict values in a time-series based network. However, my problem is that every time, I get consistently the validation loss lower than the training loss. Even at the starting of the epochs, the model always has validation loss lower than train loss which seems weird to me. Basically, I have created a subset of my Training Data (which after splitting into Validation has 42005 samples), which I use for validation (having 10502 samples). And then, there is a separate test set, which is used to evaluate the RMSE and MAPE for the network. I use a test set with 17667 samples for this very purpose.
The following outlines the essence of the network:-
# design network
model = Sequential()
model.add(LSTM(200, input_shape=(train_X.shape[1], train_X.shape[2])))
model.add(Dense(1))
model.add(Dense(1))
model.compile(loss='mae', optimizer= 'Adam') # also try mae
# fit network
es = keras.callbacks.EarlyStopping(monitor='val_loss', min_delta=0, verbose=1 , mode='min')
history = model.fit(train_X, train_y, epochs= 100, batch_size= 512 , validation_data=(val_X, val_y), verbose=2, shuffle=False, callbacks= [es])
I have also tried various batch sizes, including 128, 512, 1024, and even lower values like 10,20,40 etc., and also with only 1 dense layer or multiple dense layers, but always the test loss is lower than train and due to early stopping, the model stops at at most 5-6 epochs, which seems very weird. The loss curve for the network is accordingly shown in 
I have looked at Stackexchange, Keras Documentation and related blogs, but still, the problem isn't clear to me. Any help and suggestions on how I can resolve this would be highly appreciated.
EDIT
Used activation function 'tanh' as per:
model = Sequential()
model.add(LSTM(200, input_shape=(train_X.shape[1], train_X.shape[2])))
model.add(Dense(1, activation='tanh'))
model.add(Dense(1, activation='tanh'))
New loss curve is still weird as in :- 
machine-learning neural-network deep-learning keras lstm
asked Mar 29 at 11:21
JChatJChat
378
$endgroup$
|
show 2 more comments
$begingroup$
I am using a LSTM model to predict values in a time-series based network. However, my problem is that every time, I get consistently the validation loss lower than the training loss. Even at the starting of the epochs, the model always has validation loss lower than train loss which seems weird to me. Basically, I have created a subset of my Training Data (which after splitting into Validation has 42005 samples), which I use for validation (having 10502 samples). And then, there is a separate test set, which is used to evaluate the RMSE and MAPE for the network. I use a test set with 17667 samples for this very purpose.
The following outlines the essence of the network:-
# design network
model = Sequential()
model.add(LSTM(200, input_shape=(train_X.shape[1], train_X.shape[2])))
model.add(Dense(1))
model.add(Dense(1))
model.compile(loss='mae', optimizer= 'Adam') # also try mae
# fit network
es = keras.callbacks.EarlyStopping(monitor='val_loss', min_delta=0, verbose=1 , mode='min')
history = model.fit(train_X, train_y, epochs= 100, batch_size= 512 , validation_data=(val_X, val_y), verbose=2, shuffle=False, callbacks= [es])
I have also tried various batch sizes, including 128, 512, 1024, and even lower values like 10,20,40 etc., and also with only 1 dense layer or multiple dense layers, but always the test loss is lower than train and due to early stopping, the model stops at at most 5-6 epochs, which seems very weird. The loss curve for the network is accordingly shown in
I have looked at Stackexchange, Keras Documentation and related blogs, but still, the problem isn't clear to me. Any help and suggestions on how I can resolve this would be highly appreciated.
EDIT
Used activation function 'tanh' as per:
model = Sequential()
model.add(LSTM(200, input_shape=(train_X.shape[1], train_X.shape[2])))
model.add(Dense(1, activation='tanh'))
model.add(Dense(1, activation='tanh'))
New loss curve is still weird as in :-
machine-learning neural-network deep-learning keras lstm
asked Mar 29 at 11:21
JChatJChat
378
$endgroup$
$begingroup$
In this model, Dense layers do not have any activation function. Is usage of Linear activation intentional ?
$endgroup$
– Shamit Verma
Mar 29 at 13:55
$begingroup$
No. It's not intentional, as I was not sure of how to add dense layers, and I have added them in this fashion. Please refer to the edit, I tried this with 'tanh' activation as well, but the results are still similar.
$endgroup$
– JChat
Mar 29 at 16:36
$begingroup$
What are min and max values of output variable in train and test sets ?
$endgroup$
– Shamit Verma
Mar 29 at 17:56
$begingroup$
The output is already scaled to range 0-1 using MinMaxScaler.
$endgroup$
– JChat
Mar 29 at 19:53
$begingroup$
What's the point of the second dense layer? It also outputs just one value
$endgroup$
– Sean Owen
Mar 30 at 8:55
|
show 2 more comments
$begingroup$
I am using a LSTM model to predict values in a time-series based network. However, my problem is that every time, I get consistently the validation loss lower than the training loss. Even at the starting of the epochs, the model always has validation loss lower than train loss which seems weird to me. Basically, I have created a subset of my Training Data (which after splitting into Validation has 42005 samples), which I use for validation (having 10502 samples). And then, there is a separate test set, which is used to evaluate the RMSE and MAPE for the network. I use a test set with 17667 samples for this very purpose.
The following outlines the essence of the network:-
# design network
model = Sequential()
model.add(LSTM(200, input_shape=(train_X.shape[1], train_X.shape[2])))
model.add(Dense(1))
model.add(Dense(1))
model.compile(loss='mae', optimizer= 'Adam') # also try mae
# fit network
es = keras.callbacks.EarlyStopping(monitor='val_loss', min_delta=0, verbose=1 , mode='min')
history = model.fit(train_X, train_y, epochs= 100, batch_size= 512 , validation_data=(val_X, val_y), verbose=2, shuffle=False, callbacks= [es])
I have also tried various batch sizes, including 128, 512, 1024, and even lower values like 10,20,40 etc., and also with only 1 dense layer or multiple dense layers, but always the test loss is lower than train and due to early stopping, the model stops at at most 5-6 epochs, which seems very weird. The loss curve for the network is accordingly shown in
I have looked at Stackexchange, Keras Documentation and related blogs, but still, the problem isn't clear to me. Any help and suggestions on how I can resolve this would be highly appreciated.
EDIT
Used activation function 'tanh' as per:
model = Sequential()
model.add(LSTM(200, input_shape=(train_X.shape[1], train_X.shape[2])))
model.add(Dense(1, activation='tanh'))
model.add(Dense(1, activation='tanh'))
New loss curve is still weird as in :-
machine-learning neural-network deep-learning keras lstm
asked Mar 29 at 11:21
JChatJChat
378
$endgroup$
I am using a LSTM model to predict values in a time-series based network. However, my problem is that every time, I get consistently the validation loss lower than the training loss. Even at the starting of the epochs, the model always has validation loss lower than train loss which seems weird to me. Basically, I have created a subset of my Training Data (which after splitting into Validation has 42005 samples), which I use for validation (having 10502 samples). And then, there is a separate test set, which is used to evaluate the RMSE and MAPE for the network. I use a test set with 17667 samples for this very purpose.
The following outlines the essence of the network:-
# design network
model = Sequential()
model.add(LSTM(200, input_shape=(train_X.shape[1], train_X.shape[2])))
model.add(Dense(1))
model.add(Dense(1))
model.compile(loss='mae', optimizer= 'Adam') # also try mae
# fit network
es = keras.callbacks.EarlyStopping(monitor='val_loss', min_delta=0, verbose=1 , mode='min')
history = model.fit(train_X, train_y, epochs= 100, batch_size= 512 , validation_data=(val_X, val_y), verbose=2, shuffle=False, callbacks= [es])
I have also tried various batch sizes, including 128, 512, 1024, and even lower values like 10,20,40 etc., and also with only 1 dense layer or multiple dense layers, but always the test loss is lower than train and due to early stopping, the model stops at at most 5-6 epochs, which seems very weird. The loss curve for the network is accordingly shown in
I have looked at Stackexchange, Keras Documentation and related blogs, but still, the problem isn't clear to me. Any help and suggestions on how I can resolve this would be highly appreciated.
EDIT
Used activation function 'tanh' as per:
model = Sequential()
model.add(LSTM(200, input_shape=(train_X.shape[1], train_X.shape[2])))
model.add(Dense(1, activation='tanh'))
model.add(Dense(1, activation='tanh'))
New loss curve is still weird as in :-
machine-learning neural-network deep-learning keras lstm
machine-learning neural-network deep-learning keras lstm
asked Mar 29 at 11:21
JChatJChat
378
asked Mar 29 at 11:21
JChatJChat
378
edited Mar 29 at 16:41
JChat
asked Mar 29 at 11:21
JChatJChat
378
asked Mar 29 at 11:21
JChatJChat
378
asked Mar 29 at 11:21
JChatJChat
378
378
$begingroup$
In this model, Dense layers do not have any activation function. Is usage of Linear activation intentional ?
$endgroup$
– Shamit Verma
Mar 29 at 13:55
$begingroup$
No. It's not intentional, as I was not sure of how to add dense layers, and I have added them in this fashion. Please refer to the edit, I tried this with 'tanh' activation as well, but the results are still similar.
$endgroup$
– JChat
Mar 29 at 16:36
$begingroup$
What are min and max values of output variable in train and test sets ?
$endgroup$
– Shamit Verma
Mar 29 at 17:56
$begingroup$
The output is already scaled to range 0-1 using MinMaxScaler.
$endgroup$
– JChat
Mar 29 at 19:53
$begingroup$
What's the point of the second dense layer? It also outputs just one value
$endgroup$
– Sean Owen
Mar 30 at 8:55
|
show 2 more comments
$begingroup$
In this model, Dense layers do not have any activation function. Is usage of Linear activation intentional ?
$endgroup$
– Shamit Verma
Mar 29 at 13:55
$begingroup$
No. It's not intentional, as I was not sure of how to add dense layers, and I have added them in this fashion. Please refer to the edit, I tried this with 'tanh' activation as well, but the results are still similar.
$endgroup$
– JChat
Mar 29 at 16:36
$begingroup$
What are min and max values of output variable in train and test sets ?
$endgroup$
– Shamit Verma
Mar 29 at 17:56
$begingroup$
The output is already scaled to range 0-1 using MinMaxScaler.
$endgroup$
– JChat
Mar 29 at 19:53
$begingroup$
What's the point of the second dense layer? It also outputs just one value
$endgroup$
– Sean Owen
Mar 30 at 8:55
$begingroup$
In this model, Dense layers do not have any activation function. Is usage of Linear activation intentional ?
$endgroup$
– Shamit Verma
Mar 29 at 13:55
$begingroup$
In this model, Dense layers do not have any activation function. Is usage of Linear activation intentional ?
$endgroup$
– Shamit Verma
Mar 29 at 13:55
$begingroup$
No. It's not intentional, as I was not sure of how to add dense layers, and I have added them in this fashion. Please refer to the edit, I tried this with 'tanh' activation as well, but the results are still similar.
$endgroup$
– JChat
Mar 29 at 16:36
$begingroup$
No. It's not intentional, as I was not sure of how to add dense layers, and I have added them in this fashion. Please refer to the edit, I tried this with 'tanh' activation as well, but the results are still similar.
$endgroup$
– JChat
Mar 29 at 16:36
$begingroup$
What are min and max values of output variable in train and test sets ?
$endgroup$
– Shamit Verma
Mar 29 at 17:56
$begingroup$
What are min and max values of output variable in train and test sets ?
$endgroup$
– Shamit Verma
Mar 29 at 17:56
$begingroup$
The output is already scaled to range 0-1 using MinMaxScaler.
$endgroup$
– JChat
Mar 29 at 19:53
$begingroup$
The output is already scaled to range 0-1 using MinMaxScaler.
$endgroup$
– JChat
Mar 29 at 19:53
$begingroup$
What's the point of the second dense layer? It also outputs just one value
$endgroup$
– Sean Owen
Mar 30 at 8:55
$begingroup$
What's the point of the second dense layer? It also outputs just one value
$endgroup$
– Sean Owen
Mar 30 at 8:55
|
show 2 more comments
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function ()
return StackExchange.using("mathjaxEditing", function ()
StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix)
StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["$", "$"], ["\\(","\\)"]]);
);
);
, "mathjax-editing");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "557"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f48211%2fwhat-causes-the-network-validation-loss-to-always-be-lower-than-train-loss%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Data Science Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
Use MathJax to format equations. MathJax reference.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f48211%2fwhat-causes-the-network-validation-loss-to-always-be-lower-than-train-loss%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
$begingroup$
In this model, Dense layers do not have any activation function. Is usage of Linear activation intentional ?
$endgroup$
– Shamit Verma
Mar 29 at 13:55
$begingroup$
No. It's not intentional, as I was not sure of how to add dense layers, and I have added them in this fashion. Please refer to the edit, I tried this with 'tanh' activation as well, but the results are still similar.
$endgroup$
– JChat
Mar 29 at 16:36
$begingroup$
What are min and max values of output variable in train and test sets ?
$endgroup$
– Shamit Verma
Mar 29 at 17:56
$begingroup$
The output is already scaled to range 0-1 using MinMaxScaler.
$endgroup$
– JChat
Mar 29 at 19:53
$begingroup$
What's the point of the second dense layer? It also outputs just one value
$endgroup$
– Sean Owen
Mar 30 at 8:55