Trjtdtk

Review your own paper in Mathematics

Pre-mixing cryogenic fuels and using only one fuel tank

Why Shazam when there is already Superman?

Permission on Database

Quoting Keynes in a lecture

How could a planet have erratic days?

Why is it that I can sometimes guess the next note?

What are some good ways to treat frozen vegetables such that they behave like fresh vegetables when stir frying them?

When were female captains banned from Starfleet?

What do you call a word that can be spelled forward or backward forming two different words

Are Captain Marvel's powers affected by Thanos breaking the Tesseract and claiming the stone?

Will number of steps recorded on FitBit/any fitness tracker add up distance in PokemonGo?

Does grappling negate Mirror Image?

Merge org tables

Why do ¬, ∀ and ∃ have the same precedence?

Is it allowed to activate the ability of multiple planeswalkers in a single turn?

Why is so much work done on numerical verification of the Riemann Hypothesis?

How much theory knowledge is actually used while playing?

Stack Interview Code methods made from class Node and Smart Pointers

JIS and ISO square taper

How much of a Devil Fruit must be consumed to gain the power?

How to get directions in deep space?

Why does Carol not get rid of the Kree symbol on her suit when she changes its colours?

Does the reader need to like the PoV character?

Using SMAPE as a loss function for an LSTM

Loss function for sparse taggingRNN for classification giving vastly different results (Keras)Classifier that optimizes performance on only a subset of the data?Understanding LSTM behaviour: Validation loss smaller than training loss throughout training for regression problemExpected behaviour of loss and accuracy when using data augmentationLSTM - divide gradients by number of timesteps IMMEDIATELY or in the end?Using deep learning to classify similar imagesTuning a sequence to sequence modelIs there any standard or normal range for the amount of LSTM loss function?

1

1

$begingroup$

I am currently working on a time series forecasting problem and am looking into using an LSTM.

My final accuracy metric that I use to determine whether or not the forecast is good or not is defined as follows:

$$textSMAPE-3 = fracy_t-haty_tsum^n_t=1left(y_t+haty_tright),$$

where $y_t$ is the actual value and $haty_t$ is the forecast value. So this formula will produce a score between $[0, 1]$. This score can then be subtracted from 1 to produce a percentage accuracy of the forecast over the specified forecast horizon

$$textAccuracy = 1-fracy_t-haty_tsum^n_t=1left(y_t+haty_tright).$$

I am using keras and have seen that I can specify a custom loss function and metric. As my forecast accuracy will be measured using sMAPE 3 (as defined above) it made sense to me to use this as my loss function. This is because my networks success will be measured by how well it scores using this. A score of 0 indicates 100% accuracy and a score of 1 indicates 0% accuracy.

So this raises two questions, can a metric such as smape be used as loss function at all, if so why / why not? Are the standard loss functions within keras considered better? Perhaps because they allow quicker convergence? Does my choice of using SMAPE3 as a loss function have an impact on what kind of optimiser I should use?

I ask because I haven't seen many people using loss functions other than RMSE, MSE and MAE and there must be a reason for this.

keras time-series lstm rnn loss-function

share|improve this question

edited Nov 12 '18 at 21:33

Aesir

asked Nov 12 '18 at 11:38

AesirAesir

1498

$endgroup$

• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  This is a great question. I too have been wondering about using sMAPE. Was reading a paper on "Modeling approaches for time series forecasting and anomaly detection" (S Du, 2017) . and it mentions using sMAPE as "This metric is more robust towards outliers and it has a unified scale across different time series with different scale." (ibid., 4) But most docs I see are using the standard RMSE, MSE, and MAE metrics. Would love to have this answered.
  $endgroup$
  – JohnnyDenim
  yesterday
  

  
  
  
  

add a comment | 

1

1

$begingroup$

I am currently working on a time series forecasting problem and am looking into using an LSTM.

My final accuracy metric that I use to determine whether or not the forecast is good or not is defined as follows:

$$textSMAPE-3 = fracy_t-haty_tsum^n_t=1left(y_t+haty_tright),$$

where $y_t$ is the actual value and $haty_t$ is the forecast value. So this formula will produce a score between $[0, 1]$. This score can then be subtracted from 1 to produce a percentage accuracy of the forecast over the specified forecast horizon

$$textAccuracy = 1-fracy_t-haty_tsum^n_t=1left(y_t+haty_tright).$$

I am using keras and have seen that I can specify a custom loss function and metric. As my forecast accuracy will be measured using sMAPE 3 (as defined above) it made sense to me to use this as my loss function. This is because my networks success will be measured by how well it scores using this. A score of 0 indicates 100% accuracy and a score of 1 indicates 0% accuracy.

So this raises two questions, can a metric such as smape be used as loss function at all, if so why / why not? Are the standard loss functions within keras considered better? Perhaps because they allow quicker convergence? Does my choice of using SMAPE3 as a loss function have an impact on what kind of optimiser I should use?

I ask because I haven't seen many people using loss functions other than RMSE, MSE and MAE and there must be a reason for this.

keras time-series lstm rnn loss-function

share|improve this question

edited Nov 12 '18 at 21:33

Aesir

asked Nov 12 '18 at 11:38

AesirAesir

1498

$endgroup$

• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  This is a great question. I too have been wondering about using sMAPE. Was reading a paper on "Modeling approaches for time series forecasting and anomaly detection" (S Du, 2017) . and it mentions using sMAPE as "This metric is more robust towards outliers and it has a unified scale across different time series with different scale." (ibid., 4) But most docs I see are using the standard RMSE, MSE, and MAE metrics. Would love to have this answered.
  $endgroup$
  – JohnnyDenim
  yesterday
  

  
  
  
  

add a comment | 

$begingroup$

I am currently working on a time series forecasting problem and am looking into using an LSTM.

My final accuracy metric that I use to determine whether or not the forecast is good or not is defined as follows:

$$textSMAPE-3 = fracy_t-haty_tsum^n_t=1left(y_t+haty_tright),$$

where $y_t$ is the actual value and $haty_t$ is the forecast value. So this formula will produce a score between $[0, 1]$. This score can then be subtracted from 1 to produce a percentage accuracy of the forecast over the specified forecast horizon

$$textAccuracy = 1-fracy_t-haty_tsum^n_t=1left(y_t+haty_tright).$$

I am using keras and have seen that I can specify a custom loss function and metric. As my forecast accuracy will be measured using sMAPE 3 (as defined above) it made sense to me to use this as my loss function. This is because my networks success will be measured by how well it scores using this. A score of 0 indicates 100% accuracy and a score of 1 indicates 0% accuracy.

So this raises two questions, can a metric such as smape be used as loss function at all, if so why / why not? Are the standard loss functions within keras considered better? Perhaps because they allow quicker convergence? Does my choice of using SMAPE3 as a loss function have an impact on what kind of optimiser I should use?

I ask because I haven't seen many people using loss functions other than RMSE, MSE and MAE and there must be a reason for this.

keras time-series lstm rnn loss-function

share|improve this question

edited Nov 12 '18 at 21:33

Aesir

asked Nov 12 '18 at 11:38

AesirAesir

1498

$endgroup$

share|improve this question

edited Nov 12 '18 at 21:33

Aesir

asked Nov 12 '18 at 11:38

AesirAesir

1498

share|improve this question

edited Nov 12 '18 at 21:33

Aesir

asked Nov 12 '18 at 11:38

AesirAesir

1498

asked Nov 12 '18 at 11:38

AesirAesir

1498

asked Nov 12 '18 at 11:38

AesirAesir

1498