Does regularization penalize models that are simpler than needed? The 2019 Stack Overflow Developer Survey Results Are In Announcing the arrival of Valued Associate #679: Cesar Manara Planned maintenance scheduled April 17/18, 2019 at 00:00UTC (8:00pm US/Eastern)Different regularization parameter per parameterWhy does not ridge regression perform feature selection although it makes use of regularization?Do discriminative models overfit more than generative models?Regularization for ARIMA modelsWhy do smaller weights result in simpler models in regularization?Why regularize all parameters in the same way?What are Regularities and Regularization?Are there empirical models that predict variance?Does regularization leads to stucking in local minima?Is there a theoretical reason why simple models perform better than complex models on time series forecasting tasks?

What aspect of planet Earth must be changed to prevent the industrial revolution?

Keeping a retro style to sci-fi spaceships?

How to grep and cut numbers from a file and sum them

Would it be possible to rearrange a dragon's flight muscle to somewhat circumvent the square-cube law?

Why can't wing-mounted spoilers be used to steepen approaches?

What is special about square numbers here?

Did the UK government pay "millions and millions of dollars" to try to snag Julian Assange?

Is it ethical to upload a automatically generated paper to a non peer-reviewed site as part of a larger research?

How do I add random spotting to the same face in cycles?

ELI5: Why do they say that Israel would have been the fourth country to land a spacecraft on the Moon and why do they call it low cost?

How does ice melt when immersed in water

Typeface like Times New Roman but with "tied" percent sign

Can the prologue be the backstory of your main character?

Why did all the guest students take carriages to the Yule Ball?

I could not break this equation. Please help me

The variadic template constructor of my class cannot modify my class members, why is that so?

Derivation tree not rendering

How to test the equality of two Pearson correlation coefficients computed from the same sample?

Who or what is the being for whom Being is a question for Heidegger?

First use of “packing” as in carrying a gun

What can I do if neighbor is blocking my solar panels intentionally?

Finding the path in a graph from A to B then back to A with a minimum of shared edges

Is every episode of "Where are my Pants?" identical?

How to pronounce 1ターン?



Does regularization penalize models that are simpler than needed?



The 2019 Stack Overflow Developer Survey Results Are In
Announcing the arrival of Valued Associate #679: Cesar Manara
Planned maintenance scheduled April 17/18, 2019 at 00:00UTC (8:00pm US/Eastern)Different regularization parameter per parameterWhy does not ridge regression perform feature selection although it makes use of regularization?Do discriminative models overfit more than generative models?Regularization for ARIMA modelsWhy do smaller weights result in simpler models in regularization?Why regularize all parameters in the same way?What are Regularities and Regularization?Are there empirical models that predict variance?Does regularization leads to stucking in local minima?Is there a theoretical reason why simple models perform better than complex models on time series forecasting tasks?



.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty margin-bottom:0;








3












$begingroup$


Yes, regularization penalizes models that are more complex than needed. But does it also penalize models that are simpler than needed?






machine-learning predictive-models modeling regularization







share|cite|improve this question









$endgroup$







  • 1




    $begingroup$
    Given we use an appropriate testing procedure to select our regularisation parameter strength, it should not penalise any models unnecessarily. (+1)
    $endgroup$
    – usεr11852
    Mar 31 at 12:04

















3












$begingroup$


Yes, regularization penalizes models that are more complex than needed. But does it also penalize models that are simpler than needed?






machine-learning predictive-models modeling regularization







share|cite|improve this question









$endgroup$







  • 1




    $begingroup$
    Given we use an appropriate testing procedure to select our regularisation parameter strength, it should not penalise any models unnecessarily. (+1)
    $endgroup$
    – usεr11852
    Mar 31 at 12:04













3












3








3





$begingroup$


Yes, regularization penalizes models that are more complex than needed. But does it also penalize models that are simpler than needed?






machine-learning predictive-models modeling regularization







share|cite|improve this question









$endgroup$




Yes, regularization penalizes models that are more complex than needed. But does it also penalize models that are simpler than needed?






machine-learning predictive-models modeling regularization




machine-learning predictive-models modeling regularization






share|cite|improve this question













share|cite|improve this question











share|cite|improve this question




share|cite|improve this question










asked Mar 31 at 11:44









alienflowalienflow

275




275







  • 1




    $begingroup$
    Given we use an appropriate testing procedure to select our regularisation parameter strength, it should not penalise any models unnecessarily. (+1)
    $endgroup$
    – usεr11852
    Mar 31 at 12:04












  • 1




    $begingroup$
    Given we use an appropriate testing procedure to select our regularisation parameter strength, it should not penalise any models unnecessarily. (+1)
    $endgroup$
    – usεr11852
    Mar 31 at 12:04







1




1




$begingroup$
Given we use an appropriate testing procedure to select our regularisation parameter strength, it should not penalise any models unnecessarily. (+1)
$endgroup$
– usεr11852
Mar 31 at 12:04




$begingroup$
Given we use an appropriate testing procedure to select our regularisation parameter strength, it should not penalise any models unnecessarily. (+1)
$endgroup$
– usεr11852
Mar 31 at 12:04










1 Answer
1






active

oldest

votes


















5












$begingroup$

For regularization terms similar to $left|thetaright|_2^2$ in effect, no they don't, they only push toward simplicity, i.e. parameters closer to zero.



Error terms such as $sum_i left|y_i - f_theta(x_i)right|_2^2$ are responsible for fighting back toward complexity (penalizing over-simplification), since the simplest model, i.e. $theta = 0$, leads to a high error.



We balance these two forces by using a regularization parameter ($lambda$) in a summation like
$$frac1Nsum_i=1^N left|y_i - f_theta(x_i)right|_2^2 + lambdaleft|thetaright|_2^2,$$
where higher $lambda$ forces the model toward more simplicity.






share|cite|improve this answer











$endgroup$












  • $begingroup$
    So, regularization like L2, L1 correspond to the first case, right?
    $endgroup$
    – alienflow
    Mar 31 at 12:05






  • 1




    $begingroup$
    @alienflow yes they all force toward zero (most simple).
    $endgroup$
    – Esmailian
    Mar 31 at 12:06











Your Answer








StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "65"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);

else
createEditor();

);

function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);



);













draft saved

draft discarded


















StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstats.stackexchange.com%2fquestions%2f400388%2fdoes-regularization-penalize-models-that-are-simpler-than-needed%23new-answer', 'question_page');

);

Post as a guest















Required, but never shown

























1 Answer
1






active

oldest

votes








1 Answer
1






active

oldest

votes









active

oldest

votes






active

oldest

votes









5












$begingroup$

For regularization terms similar to $left|thetaright|_2^2$ in effect, no they don't, they only push toward simplicity, i.e. parameters closer to zero.



Error terms such as $sum_i left|y_i - f_theta(x_i)right|_2^2$ are responsible for fighting back toward complexity (penalizing over-simplification), since the simplest model, i.e. $theta = 0$, leads to a high error.



We balance these two forces by using a regularization parameter ($lambda$) in a summation like
$$frac1Nsum_i=1^N left|y_i - f_theta(x_i)right|_2^2 + lambdaleft|thetaright|_2^2,$$
where higher $lambda$ forces the model toward more simplicity.






share|cite|improve this answer











$endgroup$












  • $begingroup$
    So, regularization like L2, L1 correspond to the first case, right?
    $endgroup$
    – alienflow
    Mar 31 at 12:05






  • 1




    $begingroup$
    @alienflow yes they all force toward zero (most simple).
    $endgroup$
    – Esmailian
    Mar 31 at 12:06















5












$begingroup$

For regularization terms similar to $left|thetaright|_2^2$ in effect, no they don't, they only push toward simplicity, i.e. parameters closer to zero.



Error terms such as $sum_i left|y_i - f_theta(x_i)right|_2^2$ are responsible for fighting back toward complexity (penalizing over-simplification), since the simplest model, i.e. $theta = 0$, leads to a high error.



We balance these two forces by using a regularization parameter ($lambda$) in a summation like
$$frac1Nsum_i=1^N left|y_i - f_theta(x_i)right|_2^2 + lambdaleft|thetaright|_2^2,$$
where higher $lambda$ forces the model toward more simplicity.






share|cite|improve this answer











$endgroup$












  • $begingroup$
    So, regularization like L2, L1 correspond to the first case, right?
    $endgroup$
    – alienflow
    Mar 31 at 12:05






  • 1




    $begingroup$
    @alienflow yes they all force toward zero (most simple).
    $endgroup$
    – Esmailian
    Mar 31 at 12:06













5












5








5





$begingroup$

For regularization terms similar to $left|thetaright|_2^2$ in effect, no they don't, they only push toward simplicity, i.e. parameters closer to zero.



Error terms such as $sum_i left|y_i - f_theta(x_i)right|_2^2$ are responsible for fighting back toward complexity (penalizing over-simplification), since the simplest model, i.e. $theta = 0$, leads to a high error.



We balance these two forces by using a regularization parameter ($lambda$) in a summation like
$$frac1Nsum_i=1^N left|y_i - f_theta(x_i)right|_2^2 + lambdaleft|thetaright|_2^2,$$
where higher $lambda$ forces the model toward more simplicity.






share|cite|improve this answer











$endgroup$



For regularization terms similar to $left|thetaright|_2^2$ in effect, no they don't, they only push toward simplicity, i.e. parameters closer to zero.



Error terms such as $sum_i left|y_i - f_theta(x_i)right|_2^2$ are responsible for fighting back toward complexity (penalizing over-simplification), since the simplest model, i.e. $theta = 0$, leads to a high error.



We balance these two forces by using a regularization parameter ($lambda$) in a summation like
$$frac1Nsum_i=1^N left|y_i - f_theta(x_i)right|_2^2 + lambdaleft|thetaright|_2^2,$$
where higher $lambda$ forces the model toward more simplicity.







share|cite|improve this answer














share|cite|improve this answer



share|cite|improve this answer








edited Mar 31 at 12:10

























answered Mar 31 at 12:00









EsmailianEsmailian

42615




42615











  • $begingroup$
    So, regularization like L2, L1 correspond to the first case, right?
    $endgroup$
    – alienflow
    Mar 31 at 12:05






  • 1




    $begingroup$
    @alienflow yes they all force toward zero (most simple).
    $endgroup$
    – Esmailian
    Mar 31 at 12:06
















  • $begingroup$
    So, regularization like L2, L1 correspond to the first case, right?
    $endgroup$
    – alienflow
    Mar 31 at 12:05






  • 1




    $begingroup$
    @alienflow yes they all force toward zero (most simple).
    $endgroup$
    – Esmailian
    Mar 31 at 12:06















$begingroup$
So, regularization like L2, L1 correspond to the first case, right?
$endgroup$
– alienflow
Mar 31 at 12:05




$begingroup$
So, regularization like L2, L1 correspond to the first case, right?
$endgroup$
– alienflow
Mar 31 at 12:05




1




1




$begingroup$
@alienflow yes they all force toward zero (most simple).
$endgroup$
– Esmailian
Mar 31 at 12:06




$begingroup$
@alienflow yes they all force toward zero (most simple).
$endgroup$
– Esmailian
Mar 31 at 12:06

















draft saved

draft discarded
















































Thanks for contributing an answer to Cross Validated!


  • Please be sure to answer the question. Provide details and share your research!

But avoid …


  • Asking for help, clarification, or responding to other answers.

  • Making statements based on opinion; back them up with references or personal experience.

Use MathJax to format equations. MathJax reference.


To learn more, see our tips on writing great answers.




draft saved


draft discarded














StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstats.stackexchange.com%2fquestions%2f400388%2fdoes-regularization-penalize-models-that-are-simpler-than-needed%23new-answer', 'question_page');

);

Post as a guest















Required, but never shown





















































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown

































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown







-

Popular posts from this blog

Marja Vauras Lähteet | Aiheesta muualla | NavigointivalikkoMarja Vauras Turun yliopiston tutkimusportaalissaInfobox OKSuomalaisen Tiedeakatemian varsinaiset jäsenetKasvatustieteiden tiedekunnan dekaanit ja muu johtoMarja VaurasKoulutusvienti on kestävyys- ja ketteryyslaji (2.5.2017)laajentamallaWorldCat Identities0000 0001 0855 9405n86069603utb201588738523620927

Image
Vuonna 1953 syntyneetSuomalaiset kasvatustieteilijätSuomalaiset professoritTurun yliopiston professorit 9. maaliskuuta1953TurkuTurun yliopistonSuomalaisen tiedeakatemian Marja Vauras Henkilötiedot Syntynyt 9. maaliskuuta 1953 (ikä 66) Turku Kansallisuus Suomi Ammatti kasvatustieteen professori Koulutus ja ura Instituutti Turun yliopisto Aiheesta muualla Marja Vauras Turun yliopiston tutkimusportaalissa Infobox OK Marja Merja Sinikka Vauras (s. 9. maaliskuuta 1953 Turku) on suomalainen filosofian tohtori, ja Turun yliopiston kasvatustieteen professori. [1] [2] Hän on toiminut myös Turun yliopiston Kasvatustieteellisen tiedekunnan dekaanina. [3] Suomalaisen tiedeakatemian jäsen Vauras on ollut vuodesta 2013. [2] Lähteet | ↑ Suomen professorit 1640–2007 , Professoriliitto, 2008 (ISBN  978-952-99281-1-8 ja 978-952-99281-2-5 , viitattu 7 huhtikuuta 2019 ) ↑ a b Suomalaisen Tiedeakatemian varsinaiset jäsenet pä...
Read more

Which is better: GPT or RelGAN for text generation?2019 Community Moderator ElectionWhat is the difference between TextGAN and LM for text generation?GANs (generative adversarial networks) possible for text as well?Generator loss not decreasing- text to image synthesisChoosing a right algorithm for template-based text generationHow should I format input and output for text generation with LSTMsGumbel Softmax vs Vanilla Softmax for GAN trainingWhich neural network to choose for classification from text/speech?NLP text autoencoder that generates text in poetic meterWhat is the interpretation of the expectation notation in the GAN formulation?What is the difference between TextGAN and LM for text generation?How to prepare the data for text generation task

Image
Neighboring nodes in the network Is it canonical bit space? Why do I get two different answers for this counting problem? Python: return float 1.0 as int 1 but float 1.5 as float 1.5 Drawing a Grid/Tikz image Why is it a bad idea to hire a hitman to eliminate most corrupt politicians? Why is this clock signal connected to a capacitor to gnd? Blender 2.8 I can't see vertices, edges or faces in edit mode How to say in German "enjoying home comforts" Is there an expression that means doing something right before you will need it rather than doing it in case you might need it? Can a virus destroy the BIOS of a modern computer? Saudi Arabia Transit Visa Why "Having chlorophyll without photosynthesis is actually very dangerous" and "like living with a bomb"? What is the word for reserving something for yourself before others do? Why do bosons tend to occupy the same state? Is the Joker left-handed? What method can I use to des...
Read more

Is this part of the description of the Archfey warlock's Misty Escape feature redundant?When is entropic ward considered “used”?How does the reaction timing work for Wrath of the Storm? Can it potentially prevent the damage from the triggering attack?Does the Dark Arts Archlich warlock patrons's Arcane Invisibility activate every time you cast a level 1+ spell?When attacking while invisible, when exactly does invisibility break?Can I cast Hellish Rebuke on my turn?Do I have to “pre-cast” a reaction spell in order for it to be triggered?What happens if a Player Misty Escapes into an Invisible CreatureCan a reaction interrupt multiattack?Does the Fiend-patron warlock's Hurl Through Hell feature dispel effects that require the target to be on the same plane as the caster?What are you allowed to do while using the Warlock's Eldritch Master feature?

Image
How to Reset Passwords on Multiple Websites Easily? How can a function with a hole (removable discontinuity) equal a function with no hole? What is the opposite of 'gravitas'? Return the Closest Prime Number Term for the "extreme-extension" version of a straw man fallacy? How to check is there any negative term in a large list? How does it work when somebody invests in my business? How do I rename a Linux host without needing to reboot for the rename to take effect? Avoiding estate tax by giving multiple gifts Purchasing a ticket for someone else in another country? Energy of the particles in the particle accelerator How can I get through very long and very dry, but also very useful technical documents when learning a new tool? How did Doctor Strange see the winning outcome in Avengers: Infinity War? How to create a 32-bit integer from eight (8) 4-bit integers? What can we do to stop prior company from asking us questions? How to be diplo...
Read more