Trjtdtk

Is there a concise way to say "all of the X, one of each"?

List *all* the tuples!

How to deal with a team lead who never gives me credit?

Check which numbers satisfy the condition [A*B*C = A! + B! + C!]

Stars Make Stars

Is there a service that would inform me whenever a new direct route is scheduled from a given airport?

ListPlot join points by nearest neighbor rather than order

Dominant seventh chord in the major scale contains diminished triad of the seventh?

Models of set theory where not every set can be linearly ordered

Is there a documented rationale why the House Ways and Means chairman can demand tax info?

How can I fade player when goes inside or outside of the area?

Output the ŋarâþ crîþ alphabet song without using (m)any letters

Is it ethical to give a final exam after the professor has quit before teaching the remaining chapters of the course?

Sorting numerically

How to bypass password on Windows XP account?

Can inflation occur in a positive-sum game currency system such as the Stack Exchange reputation system?

What's the purpose of writing one's academic bio in 3rd person?

Gastric acid as a weapon

Does accepting a pardon have any bearing on trying that person for the same crime in a sovereign jurisdiction?

Why are there no cargo aircraft with "flying wing" design?

How do I stop a creek from eroding my steep embankment?

If Jon Snow became King of the Seven Kingdoms what would his regnal number be?

Letter Boxed validator

Is above average number of years spent on PhD considered a red flag in future academia or industry positions?

Regularization: global or layerwise?

0

$begingroup$

Keras gives you the option to apply regularization differently to different layers. I mean, why not? Though when I first learned about neural nets (from ESL), I thought of it as a global parameter.

Global is simpler to tune, but obviously a global penalty can be no better than equally efficient when compared to some optimal set of layerwise ones.

So, what are the cases where different penalties for different layers will work better than a single global penalty, and better-enough to be worth the bother?

machine-learning neural-network keras regularization

share|improve this question

asked Apr 1 at 22:28

generic_usergeneric_user

30418

$endgroup$

add a comment | 

0

$begingroup$

Keras gives you the option to apply regularization differently to different layers. I mean, why not? Though when I first learned about neural nets (from ESL), I thought of it as a global parameter.

Global is simpler to tune, but obviously a global penalty can be no better than equally efficient when compared to some optimal set of layerwise ones.

So, what are the cases where different penalties for different layers will work better than a single global penalty, and better-enough to be worth the bother?

machine-learning neural-network keras regularization

share|improve this question

asked Apr 1 at 22:28

generic_usergeneric_user

30418

$endgroup$

add a comment | 

$begingroup$

Keras gives you the option to apply regularization differently to different layers. I mean, why not? Though when I first learned about neural nets (from ESL), I thought of it as a global parameter.

Global is simpler to tune, but obviously a global penalty can be no better than equally efficient when compared to some optimal set of layerwise ones.

So, what are the cases where different penalties for different layers will work better than a single global penalty, and better-enough to be worth the bother?

machine-learning neural-network keras regularization

share|improve this question

asked Apr 1 at 22:28

generic_usergeneric_user

30418

$endgroup$

share|improve this question

asked Apr 1 at 22:28

generic_usergeneric_user

30418

share|improve this question

asked Apr 1 at 22:28

generic_usergeneric_user

30418

asked Apr 1 at 22:28

generic_usergeneric_user

30418

asked Apr 1 at 22:28

generic_usergeneric_user

30418

1 Answer
1

active

oldest

votes

0

$begingroup$

Regularisation is a technique to solve overfitting.

This feature from Keras is going to help a lot in many scenarios. Few times we don't want simpler but a granulr tuning.

1. CNN: we all know that each convolution layer can contribute to certain set of features from the dataset, and we now a days know what it is trying to do, by defining regularisation to each layer differently, we can better understand how each layer is effecting the final output


2. Transfer Learning: Where we want learn from the already trained network, and use that domain knowledge. now during this, we can now control, how much we want to regularise before/after merging from the base network.


3. Multi Task Learning: This is a technique in which we learn multiple tasks together, now with this kind of regularisation we can now control before the merge of the layers, how much of the information can be merged.

these are the quick things i could think of. But there are definitely lots of other uses.

Vote up, if this helps ;)

share|improve this answer

answered Apr 1 at 23:46

William ScottWilliam Scott

1063

$endgroup$

• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  Are you a neural network?
  $endgroup$
  – generic_user
  Apr 1 at 23:55
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  You are really a generic_user. Lol.
  $endgroup$
  – William Scott
  Apr 2 at 0:26
  

  
  
  
  

add a comment | 

Your Answer

StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "557"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);

else
createEditor();

);

function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);



);

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);

Sign up using Google

Sign up using Facebook

Sign up using Email and Password

Post as a guest

Name

Email

Required, but never shown

StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f48384%2fregularization-global-or-layerwise%23new-answer', 'question_page');

);

Post as a guest

Name

Email

Required, but never shown

0

$begingroup$

Regularisation is a technique to solve overfitting.

This feature from Keras is going to help a lot in many scenarios. Few times we don't want simpler but a granulr tuning.

1. CNN: we all know that each convolution layer can contribute to certain set of features from the dataset, and we now a days know what it is trying to do, by defining regularisation to each layer differently, we can better understand how each layer is effecting the final output


2. Transfer Learning: Where we want learn from the already trained network, and use that domain knowledge. now during this, we can now control, how much we want to regularise before/after merging from the base network.


3. Multi Task Learning: This is a technique in which we learn multiple tasks together, now with this kind of regularisation we can now control before the merge of the layers, how much of the information can be merged.

these are the quick things i could think of. But there are definitely lots of other uses.

Vote up, if this helps ;)

share|improve this answer

answered Apr 1 at 23:46

William ScottWilliam Scott

1063

$endgroup$

• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  Are you a neural network?
  $endgroup$
  – generic_user
  Apr 1 at 23:55
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  You are really a generic_user. Lol.
  $endgroup$
  – William Scott
  Apr 2 at 0:26
  

  
  
  
  

add a comment | 

0

$begingroup$

Regularisation is a technique to solve overfitting.

This feature from Keras is going to help a lot in many scenarios. Few times we don't want simpler but a granulr tuning.

1. CNN: we all know that each convolution layer can contribute to certain set of features from the dataset, and we now a days know what it is trying to do, by defining regularisation to each layer differently, we can better understand how each layer is effecting the final output


2. Transfer Learning: Where we want learn from the already trained network, and use that domain knowledge. now during this, we can now control, how much we want to regularise before/after merging from the base network.


3. Multi Task Learning: This is a technique in which we learn multiple tasks together, now with this kind of regularisation we can now control before the merge of the layers, how much of the information can be merged.

these are the quick things i could think of. But there are definitely lots of other uses.

Vote up, if this helps ;)

share|improve this answer

answered Apr 1 at 23:46

William ScottWilliam Scott

1063

$endgroup$

• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  Are you a neural network?
  $endgroup$
  – generic_user
  Apr 1 at 23:55
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  You are really a generic_user. Lol.
  $endgroup$
  – William Scott
  Apr 2 at 0:26
  

  
  
  
  

add a comment | 

$begingroup$

Regularisation is a technique to solve overfitting.

This feature from Keras is going to help a lot in many scenarios. Few times we don't want simpler but a granulr tuning.

1. CNN: we all know that each convolution layer can contribute to certain set of features from the dataset, and we now a days know what it is trying to do, by defining regularisation to each layer differently, we can better understand how each layer is effecting the final output


2. Transfer Learning: Where we want learn from the already trained network, and use that domain knowledge. now during this, we can now control, how much we want to regularise before/after merging from the base network.


3. Multi Task Learning: This is a technique in which we learn multiple tasks together, now with this kind of regularisation we can now control before the merge of the layers, how much of the information can be merged.

these are the quick things i could think of. But there are definitely lots of other uses.

Vote up, if this helps ;)

share|improve this answer

answered Apr 1 at 23:46

William ScottWilliam Scott

1063

$endgroup$

share|improve this answer

answered Apr 1 at 23:46

William ScottWilliam Scott

1063

answered Apr 1 at 23:46

William ScottWilliam Scott

1063

answered Apr 1 at 23:46

William ScottWilliam Scott

1063