Trjtdtk

How did the USSR manage to innovate in an environment characterized by government censorship and high bureaucracy?

Either or Neither in sentence with another negative

Why do I get two different answers for this counting problem?

Why is consensus so controversial in Britain?

Why not use SQL instead of GraphQL?

What is the word for reserving something for yourself before others do?

The use of multiple foreign keys on same column in SQL Server

can i play a electric guitar through a bass amp?

US citizen flying to France today and my passport expires in less than 2 months

What defenses are there against being summoned by the Gate spell?

Today is the Center

Font hinting is lost in Chrome-like browsers (for some languages )

Why are 150k or 200k jobs considered good when there are 300k+ births a month?

What's the output of a record cartridge playing an out-of-speed record

Risk of getting Chronic Wasting Disease (CWD) in the United States?

Mathematical cryptic clues

Do I have a twin with permutated remainders?

What is the offset in a seaplane's hull?

Email Account under attack (really) - anything I can do?

Pattern match does not work in bash script

Is it tax fraud for an individual to declare non-taxable revenue as taxable income? (US tax laws)

Smoothness of finite-dimensional functional calculus

Why do falling prices hurt debtors?

What would happen to a modern skyscraper if it rains micro blackholes?

Detecting overfitting in deep learning in a regression task

0

$begingroup$

I have regression task at hand, but I am not sure how to consistently detect/prevent overfitting.

As my code's currently written, it will save the model on each epoch as long as

• the validation loss is lower than a tracked min_valid_loss;


• the difference between the training loss and the validation loss is not more than threshold, which is an arbitrary number.

The latter is necessary, I feel, to ensure that the model is not saved when the loss curves start diverging too much (particularly in case where training loss keeps decreasing but validation loss starts to increase). My issue is that I don't like the arbitrariness of this number. Let's say that I put threshold to 0.2, then the model will be saved as long as the difference between training and validation loss is less than 0.2. But that value is not based on anything.

Are there any guidelines to deal with this issue?

As a verbose, incomplete example of what the current code looks like (PyTorch, ReduceLROnPlateau scheduler):

threshold = 0.2
valid_loss_min = np.inf
for e in range(epochs):
 # list of losses (one per batch)
 train_losses = train(...)
 valid_losses = validate(...)

 train_loss = np.mean(train_losses)
 valid_loss = np.mean(valid_losses)

 if valid_loss <= valid_loss_min:
 if valid_loss - train_loss < threshold:
 torch.save(model.state_dict(), 'checkpoint.pth')

 scheduler.step(valid_loss)

deep-learning regression overfitting

share|improve this question

edited Mar 27 at 13:51

Bram Vanroy

asked Mar 27 at 13:37

Bram VanroyBram Vanroy

359

$endgroup$

• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  there is no such standard to tell you what is a good threshold between training and validation accuracy that tells the model is not overfitted, the smaller gap between these two is better, for example, 1%, 2%, 3% like that.
  $endgroup$
  – honar.cs
  Mar 27 at 16:47
  

  
  
  
  

add a comment | 

0

$begingroup$

I have regression task at hand, but I am not sure how to consistently detect/prevent overfitting.

As my code's currently written, it will save the model on each epoch as long as

• the validation loss is lower than a tracked min_valid_loss;


• the difference between the training loss and the validation loss is not more than threshold, which is an arbitrary number.

The latter is necessary, I feel, to ensure that the model is not saved when the loss curves start diverging too much (particularly in case where training loss keeps decreasing but validation loss starts to increase). My issue is that I don't like the arbitrariness of this number. Let's say that I put threshold to 0.2, then the model will be saved as long as the difference between training and validation loss is less than 0.2. But that value is not based on anything.

Are there any guidelines to deal with this issue?

As a verbose, incomplete example of what the current code looks like (PyTorch, ReduceLROnPlateau scheduler):

threshold = 0.2
valid_loss_min = np.inf
for e in range(epochs):
 # list of losses (one per batch)
 train_losses = train(...)
 valid_losses = validate(...)

 train_loss = np.mean(train_losses)
 valid_loss = np.mean(valid_losses)

 if valid_loss <= valid_loss_min:
 if valid_loss - train_loss < threshold:
 torch.save(model.state_dict(), 'checkpoint.pth')

 scheduler.step(valid_loss)

deep-learning regression overfitting

share|improve this question

edited Mar 27 at 13:51

Bram Vanroy

asked Mar 27 at 13:37

Bram VanroyBram Vanroy

359

$endgroup$

• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  there is no such standard to tell you what is a good threshold between training and validation accuracy that tells the model is not overfitted, the smaller gap between these two is better, for example, 1%, 2%, 3% like that.
  $endgroup$
  – honar.cs
  Mar 27 at 16:47
  

  
  
  
  

add a comment | 

$begingroup$

I have regression task at hand, but I am not sure how to consistently detect/prevent overfitting.

As my code's currently written, it will save the model on each epoch as long as

• the validation loss is lower than a tracked min_valid_loss;


• the difference between the training loss and the validation loss is not more than threshold, which is an arbitrary number.

The latter is necessary, I feel, to ensure that the model is not saved when the loss curves start diverging too much (particularly in case where training loss keeps decreasing but validation loss starts to increase). My issue is that I don't like the arbitrariness of this number. Let's say that I put threshold to 0.2, then the model will be saved as long as the difference between training and validation loss is less than 0.2. But that value is not based on anything.

Are there any guidelines to deal with this issue?

As a verbose, incomplete example of what the current code looks like (PyTorch, ReduceLROnPlateau scheduler):

threshold = 0.2
valid_loss_min = np.inf
for e in range(epochs):
 # list of losses (one per batch)
 train_losses = train(...)
 valid_losses = validate(...)

 train_loss = np.mean(train_losses)
 valid_loss = np.mean(valid_losses)

 if valid_loss <= valid_loss_min:
 if valid_loss - train_loss < threshold:
 torch.save(model.state_dict(), 'checkpoint.pth')

 scheduler.step(valid_loss)

deep-learning regression overfitting

share|improve this question

edited Mar 27 at 13:51

Bram Vanroy

asked Mar 27 at 13:37

Bram VanroyBram Vanroy

359

$endgroup$

threshold = 0.2
valid_loss_min = np.inf
for e in range(epochs):
 # list of losses (one per batch)
 train_losses = train(...)
 valid_losses = validate(...)

 train_loss = np.mean(train_losses)
 valid_loss = np.mean(valid_losses)

 if valid_loss <= valid_loss_min:
 if valid_loss - train_loss < threshold:
 torch.save(model.state_dict(), 'checkpoint.pth')

 scheduler.step(valid_loss)

share|improve this question

edited Mar 27 at 13:51

Bram Vanroy

asked Mar 27 at 13:37

Bram VanroyBram Vanroy

359

share|improve this question

edited Mar 27 at 13:51

Bram Vanroy

asked Mar 27 at 13:37

Bram VanroyBram Vanroy

359

asked Mar 27 at 13:37

Bram VanroyBram Vanroy

359

asked Mar 27 at 13:37

Bram VanroyBram Vanroy

359