Does regularization penalize models that are simpler than needed? The 2019 Stack Overflow Developer Survey Results Are In Announcing the arrival of Valued Associate #679: Cesar Manara Planned maintenance scheduled April 17/18, 2019 at 00:00UTC (8:00pm US/Eastern)Different regularization parameter per parameterWhy does not ridge regression perform feature selection although it makes use of regularization?Do discriminative models overfit more than generative models?Regularization for ARIMA modelsWhy do smaller weights result in simpler models in regularization?Why regularize all parameters in the same way?What are Regularities and Regularization?Are there empirical models that predict variance?Does regularization leads to stucking in local minima?Is there a theoretical reason why simple models perform better than complex models on time series forecasting tasks?

What aspect of planet Earth must be changed to prevent the industrial revolution?

Keeping a retro style to sci-fi spaceships?

How to grep and cut numbers from a file and sum them

Would it be possible to rearrange a dragon's flight muscle to somewhat circumvent the square-cube law?

Why can't wing-mounted spoilers be used to steepen approaches?

What is special about square numbers here?

Did the UK government pay "millions and millions of dollars" to try to snag Julian Assange?

Is it ethical to upload a automatically generated paper to a non peer-reviewed site as part of a larger research?

How do I add random spotting to the same face in cycles?

ELI5: Why do they say that Israel would have been the fourth country to land a spacecraft on the Moon and why do they call it low cost?

How does ice melt when immersed in water

Typeface like Times New Roman but with "tied" percent sign

Can the prologue be the backstory of your main character?

Why did all the guest students take carriages to the Yule Ball?

I could not break this equation. Please help me

The variadic template constructor of my class cannot modify my class members, why is that so?

Derivation tree not rendering

How to test the equality of two Pearson correlation coefficients computed from the same sample?

Who or what is the being for whom Being is a question for Heidegger?

First use of “packing” as in carrying a gun

What can I do if neighbor is blocking my solar panels intentionally?

Finding the path in a graph from A to B then back to A with a minimum of shared edges

Is every episode of "Where are my Pants?" identical?

How to pronounce 1ターン?



Does regularization penalize models that are simpler than needed?



The 2019 Stack Overflow Developer Survey Results Are In
Announcing the arrival of Valued Associate #679: Cesar Manara
Planned maintenance scheduled April 17/18, 2019 at 00:00UTC (8:00pm US/Eastern)Different regularization parameter per parameterWhy does not ridge regression perform feature selection although it makes use of regularization?Do discriminative models overfit more than generative models?Regularization for ARIMA modelsWhy do smaller weights result in simpler models in regularization?Why regularize all parameters in the same way?What are Regularities and Regularization?Are there empirical models that predict variance?Does regularization leads to stucking in local minima?Is there a theoretical reason why simple models perform better than complex models on time series forecasting tasks?



.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty margin-bottom:0;








3












$begingroup$


Yes, regularization penalizes models that are more complex than needed. But does it also penalize models that are simpler than needed?






machine-learning predictive-models modeling regularization







share|cite|improve this question









$endgroup$







  • 1




    $begingroup$
    Given we use an appropriate testing procedure to select our regularisation parameter strength, it should not penalise any models unnecessarily. (+1)
    $endgroup$
    – usεr11852
    Mar 31 at 12:04

















3












$begingroup$


Yes, regularization penalizes models that are more complex than needed. But does it also penalize models that are simpler than needed?






machine-learning predictive-models modeling regularization







share|cite|improve this question









$endgroup$







  • 1




    $begingroup$
    Given we use an appropriate testing procedure to select our regularisation parameter strength, it should not penalise any models unnecessarily. (+1)
    $endgroup$
    – usεr11852
    Mar 31 at 12:04













3












3








3





$begingroup$


Yes, regularization penalizes models that are more complex than needed. But does it also penalize models that are simpler than needed?






machine-learning predictive-models modeling regularization







share|cite|improve this question









$endgroup$




Yes, regularization penalizes models that are more complex than needed. But does it also penalize models that are simpler than needed?






machine-learning predictive-models modeling regularization




machine-learning predictive-models modeling regularization






share|cite|improve this question













share|cite|improve this question











share|cite|improve this question




share|cite|improve this question










asked Mar 31 at 11:44









alienflowalienflow

275




275







  • 1




    $begingroup$
    Given we use an appropriate testing procedure to select our regularisation parameter strength, it should not penalise any models unnecessarily. (+1)
    $endgroup$
    – usεr11852
    Mar 31 at 12:04












  • 1




    $begingroup$
    Given we use an appropriate testing procedure to select our regularisation parameter strength, it should not penalise any models unnecessarily. (+1)
    $endgroup$
    – usεr11852
    Mar 31 at 12:04







1




1




$begingroup$
Given we use an appropriate testing procedure to select our regularisation parameter strength, it should not penalise any models unnecessarily. (+1)
$endgroup$
– usεr11852
Mar 31 at 12:04




$begingroup$
Given we use an appropriate testing procedure to select our regularisation parameter strength, it should not penalise any models unnecessarily. (+1)
$endgroup$
– usεr11852
Mar 31 at 12:04










1 Answer
1






active

oldest

votes


















5












$begingroup$

For regularization terms similar to $left|thetaright|_2^2$ in effect, no they don't, they only push toward simplicity, i.e. parameters closer to zero.



Error terms such as $sum_i left|y_i - f_theta(x_i)right|_2^2$ are responsible for fighting back toward complexity (penalizing over-simplification), since the simplest model, i.e. $theta = 0$, leads to a high error.



We balance these two forces by using a regularization parameter ($lambda$) in a summation like
$$frac1Nsum_i=1^N left|y_i - f_theta(x_i)right|_2^2 + lambdaleft|thetaright|_2^2,$$
where higher $lambda$ forces the model toward more simplicity.






share|cite|improve this answer











$endgroup$












  • $begingroup$
    So, regularization like L2, L1 correspond to the first case, right?
    $endgroup$
    – alienflow
    Mar 31 at 12:05






  • 1




    $begingroup$
    @alienflow yes they all force toward zero (most simple).
    $endgroup$
    – Esmailian
    Mar 31 at 12:06











Your Answer








StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "65"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);

else
createEditor();

);

function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);



);













draft saved

draft discarded


















StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstats.stackexchange.com%2fquestions%2f400388%2fdoes-regularization-penalize-models-that-are-simpler-than-needed%23new-answer', 'question_page');

);

Post as a guest















Required, but never shown

























1 Answer
1






active

oldest

votes








1 Answer
1






active

oldest

votes









active

oldest

votes






active

oldest

votes









5












$begingroup$

For regularization terms similar to $left|thetaright|_2^2$ in effect, no they don't, they only push toward simplicity, i.e. parameters closer to zero.



Error terms such as $sum_i left|y_i - f_theta(x_i)right|_2^2$ are responsible for fighting back toward complexity (penalizing over-simplification), since the simplest model, i.e. $theta = 0$, leads to a high error.



We balance these two forces by using a regularization parameter ($lambda$) in a summation like
$$frac1Nsum_i=1^N left|y_i - f_theta(x_i)right|_2^2 + lambdaleft|thetaright|_2^2,$$
where higher $lambda$ forces the model toward more simplicity.






share|cite|improve this answer











$endgroup$












  • $begingroup$
    So, regularization like L2, L1 correspond to the first case, right?
    $endgroup$
    – alienflow
    Mar 31 at 12:05






  • 1




    $begingroup$
    @alienflow yes they all force toward zero (most simple).
    $endgroup$
    – Esmailian
    Mar 31 at 12:06















5












$begingroup$

For regularization terms similar to $left|thetaright|_2^2$ in effect, no they don't, they only push toward simplicity, i.e. parameters closer to zero.



Error terms such as $sum_i left|y_i - f_theta(x_i)right|_2^2$ are responsible for fighting back toward complexity (penalizing over-simplification), since the simplest model, i.e. $theta = 0$, leads to a high error.



We balance these two forces by using a regularization parameter ($lambda$) in a summation like
$$frac1Nsum_i=1^N left|y_i - f_theta(x_i)right|_2^2 + lambdaleft|thetaright|_2^2,$$
where higher $lambda$ forces the model toward more simplicity.






share|cite|improve this answer











$endgroup$












  • $begingroup$
    So, regularization like L2, L1 correspond to the first case, right?
    $endgroup$
    – alienflow
    Mar 31 at 12:05






  • 1




    $begingroup$
    @alienflow yes they all force toward zero (most simple).
    $endgroup$
    – Esmailian
    Mar 31 at 12:06













5












5








5





$begingroup$

For regularization terms similar to $left|thetaright|_2^2$ in effect, no they don't, they only push toward simplicity, i.e. parameters closer to zero.



Error terms such as $sum_i left|y_i - f_theta(x_i)right|_2^2$ are responsible for fighting back toward complexity (penalizing over-simplification), since the simplest model, i.e. $theta = 0$, leads to a high error.



We balance these two forces by using a regularization parameter ($lambda$) in a summation like
$$frac1Nsum_i=1^N left|y_i - f_theta(x_i)right|_2^2 + lambdaleft|thetaright|_2^2,$$
where higher $lambda$ forces the model toward more simplicity.






share|cite|improve this answer











$endgroup$



For regularization terms similar to $left|thetaright|_2^2$ in effect, no they don't, they only push toward simplicity, i.e. parameters closer to zero.



Error terms such as $sum_i left|y_i - f_theta(x_i)right|_2^2$ are responsible for fighting back toward complexity (penalizing over-simplification), since the simplest model, i.e. $theta = 0$, leads to a high error.



We balance these two forces by using a regularization parameter ($lambda$) in a summation like
$$frac1Nsum_i=1^N left|y_i - f_theta(x_i)right|_2^2 + lambdaleft|thetaright|_2^2,$$
where higher $lambda$ forces the model toward more simplicity.







share|cite|improve this answer














share|cite|improve this answer



share|cite|improve this answer








edited Mar 31 at 12:10

























answered Mar 31 at 12:00









EsmailianEsmailian

42615




42615











  • $begingroup$
    So, regularization like L2, L1 correspond to the first case, right?
    $endgroup$
    – alienflow
    Mar 31 at 12:05






  • 1




    $begingroup$
    @alienflow yes they all force toward zero (most simple).
    $endgroup$
    – Esmailian
    Mar 31 at 12:06
















  • $begingroup$
    So, regularization like L2, L1 correspond to the first case, right?
    $endgroup$
    – alienflow
    Mar 31 at 12:05






  • 1




    $begingroup$
    @alienflow yes they all force toward zero (most simple).
    $endgroup$
    – Esmailian
    Mar 31 at 12:06















$begingroup$
So, regularization like L2, L1 correspond to the first case, right?
$endgroup$
– alienflow
Mar 31 at 12:05




$begingroup$
So, regularization like L2, L1 correspond to the first case, right?
$endgroup$
– alienflow
Mar 31 at 12:05




1




1




$begingroup$
@alienflow yes they all force toward zero (most simple).
$endgroup$
– Esmailian
Mar 31 at 12:06




$begingroup$
@alienflow yes they all force toward zero (most simple).
$endgroup$
– Esmailian
Mar 31 at 12:06

















draft saved

draft discarded
















































Thanks for contributing an answer to Cross Validated!


  • Please be sure to answer the question. Provide details and share your research!

But avoid …


  • Asking for help, clarification, or responding to other answers.

  • Making statements based on opinion; back them up with references or personal experience.

Use MathJax to format equations. MathJax reference.


To learn more, see our tips on writing great answers.




draft saved


draft discarded














StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstats.stackexchange.com%2fquestions%2f400388%2fdoes-regularization-penalize-models-that-are-simpler-than-needed%23new-answer', 'question_page');

);

Post as a guest















Required, but never shown





















































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown

































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown







-

Popular posts from this blog

Adding axes to figuresAdding axes labels to LaTeX figuresLaTeX equivalent of ConTeXt buffersRotate a node but not its content: the case of the ellipse decorationHow to define the default vertical distance between nodes?TikZ scaling graphic and adjust node position and keep font sizeNumerical conditional within tikz keys?adding axes to shapesAlign axes across subfiguresAdding figures with a certain orderLine up nested tikz enviroments or how to get rid of themAdding axes labels to LaTeX figures

Image
Fear of getting stuck on one programming language / technology that is not used in my country Is there a single word describing earning money through any means? Why do we read the Megillah by night and by day? Pre-mixing cryogenic fuels and using only one fuel tank Is the U.S. Code copyrighted by the Government? Should I outline or discovery write my stories? What is this called? Old film camera viewer? Does the expansion of the universe explain why the universe doesn't collapse? Removing files under particular conditions (number of files, file age) Argument list too long when zipping large list of certain files in a folder When a Cleric spontaneously casts a Cure Light Wounds spell, will a Pearl of Power recover the original spell or Cure Light Wounds? Non-trope happy ending? Is it better practice to read straight from sheet music rather than memorize it? Do Legal Documents Require Signing In Standard Pen Colors? Does an advisor owe his/her student anyt...
Read more

Tähtien Talli Jäsenet | Lähteet | NavigointivalikkoSuomen Hippos – Tähtien Talli

Image
RaviurheiluUrheilun kunniagalleriat Suomen HippoksenravihevostenLämminverisiäsuomenhevosiaSuomen raviurheilunBrad de Veluwe Tähtien Talli on vuonna 1995 perustettu Suomen Hippoksen ylläpitämä suomalaisten ravihevosten kunniagalleria. Tähtien Tallissa on toistaiseksi yhteensä 20 hevosta. Lämminverisiä galleriassa on kymmenen ja suomenhevosia kymmenen. Ulkomailla syntyneitä hevosia on mukana kolme. Vaikka Suomen raviurheilun perinteet ulottuvat jo 1800-luvun puolelle, painottuvat Tähtien Tallien valinnat viimeisten 20-30 vuoden aikana kilpailleisiin hevosiin. Viimeisin kunnianosoituksen saanut hevonen on Brad de Veluwe joka liitettiin Tähtien Talliin vuoden 2015 Ravigaalassa. Jäsenet | Reipas vuoden 1958 Kuninkuusraveissa Lahdessa. Passionate Kemp lämminveristen Suomen mestaruuden voittoloimi yllään. Hevonen Kilpailuvuodet Kasvattaja Valinta- vuosi Charme Asserdal 1975–1981 Erik Andersson 1995...
Read more

Do these cracks on my tires look bad? The Next CEO of Stack OverflowDry rot tire should I replace?Having to replace tiresFishtailed so easily? Bad tires? ABS?Filling the tires with something other than air, to avoid puncture hassles?Used Michelin tires safe to install?Do these tyre cracks necessitate replacement?Rumbling noise: tires or mechanicalIs it possible to fix noisy feathered tires?Are bad winter tires still better than summer tires in winter?Torque converter failure - Related to replacing only 2 tires?Why use snow tires on all 4 wheels on 2-wheel-drive cars?

Image
Small nick on power cord from an electric alarm clock, and copper wiring exposed but intact Why do we say “un seul M” and not “une seule M” even though M is a “consonne”? Read/write a pipe-delimited file line by line with some simple text manipulation Free fall ellipse or parabola? Why did the Drakh emissary look so blurred in S04:E11 "Lines of Communication"? A hang glider, sudden unexpected lift to 25,000 feet altitude, what could do this? Planeswalker Ability and Death Timing Which acid/base does a strong base/acid react when added to a buffer solution? Is the 21st century's idea of "freedom of speech" based on precedent? Shortening a title without changing its meaning Man transported from Alternate World into ours by a Neutrino Detector Is it reasonable to ask other researchers to send me their previous grant applications? Find a path from s to t using as few red nodes as possible What difference does it make matching a w...
Read more