Does regularization penalize models that are simpler than needed? The 2019 Stack Overflow Developer Survey Results Are In Announcing the arrival of Valued Associate #679: Cesar Manara Planned maintenance scheduled April 17/18, 2019 at 00:00UTC (8:00pm US/Eastern)Different regularization parameter per parameterWhy does not ridge regression perform feature selection although it makes use of regularization?Do discriminative models overfit more than generative models?Regularization for ARIMA modelsWhy do smaller weights result in simpler models in regularization?Why regularize all parameters in the same way?What are Regularities and Regularization?Are there empirical models that predict variance?Does regularization leads to stucking in local minima?Is there a theoretical reason why simple models perform better than complex models on time series forecasting tasks?

What aspect of planet Earth must be changed to prevent the industrial revolution?

Keeping a retro style to sci-fi spaceships?

How to grep and cut numbers from a file and sum them

Would it be possible to rearrange a dragon's flight muscle to somewhat circumvent the square-cube law?

Why can't wing-mounted spoilers be used to steepen approaches?

What is special about square numbers here?

Did the UK government pay "millions and millions of dollars" to try to snag Julian Assange?

Is it ethical to upload a automatically generated paper to a non peer-reviewed site as part of a larger research?

How do I add random spotting to the same face in cycles?

ELI5: Why do they say that Israel would have been the fourth country to land a spacecraft on the Moon and why do they call it low cost?

How does ice melt when immersed in water

Typeface like Times New Roman but with "tied" percent sign

Can the prologue be the backstory of your main character?

Why did all the guest students take carriages to the Yule Ball?

I could not break this equation. Please help me

The variadic template constructor of my class cannot modify my class members, why is that so?

Derivation tree not rendering

How to test the equality of two Pearson correlation coefficients computed from the same sample?

Who or what is the being for whom Being is a question for Heidegger?

First use of “packing” as in carrying a gun

What can I do if neighbor is blocking my solar panels intentionally?

Finding the path in a graph from A to B then back to A with a minimum of shared edges

Is every episode of "Where are my Pants?" identical?

How to pronounce 1ターン?



Does regularization penalize models that are simpler than needed?



The 2019 Stack Overflow Developer Survey Results Are In
Announcing the arrival of Valued Associate #679: Cesar Manara
Planned maintenance scheduled April 17/18, 2019 at 00:00UTC (8:00pm US/Eastern)Different regularization parameter per parameterWhy does not ridge regression perform feature selection although it makes use of regularization?Do discriminative models overfit more than generative models?Regularization for ARIMA modelsWhy do smaller weights result in simpler models in regularization?Why regularize all parameters in the same way?What are Regularities and Regularization?Are there empirical models that predict variance?Does regularization leads to stucking in local minima?Is there a theoretical reason why simple models perform better than complex models on time series forecasting tasks?



.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty margin-bottom:0;








3












$begingroup$


Yes, regularization penalizes models that are more complex than needed. But does it also penalize models that are simpler than needed?






machine-learning predictive-models modeling regularization







share|cite|improve this question









$endgroup$







  • 1




    $begingroup$
    Given we use an appropriate testing procedure to select our regularisation parameter strength, it should not penalise any models unnecessarily. (+1)
    $endgroup$
    – usεr11852
    Mar 31 at 12:04

















3












$begingroup$


Yes, regularization penalizes models that are more complex than needed. But does it also penalize models that are simpler than needed?






machine-learning predictive-models modeling regularization







share|cite|improve this question









$endgroup$







  • 1




    $begingroup$
    Given we use an appropriate testing procedure to select our regularisation parameter strength, it should not penalise any models unnecessarily. (+1)
    $endgroup$
    – usεr11852
    Mar 31 at 12:04













3












3








3





$begingroup$


Yes, regularization penalizes models that are more complex than needed. But does it also penalize models that are simpler than needed?






machine-learning predictive-models modeling regularization







share|cite|improve this question









$endgroup$




Yes, regularization penalizes models that are more complex than needed. But does it also penalize models that are simpler than needed?






machine-learning predictive-models modeling regularization




machine-learning predictive-models modeling regularization






share|cite|improve this question













share|cite|improve this question











share|cite|improve this question




share|cite|improve this question










asked Mar 31 at 11:44









alienflowalienflow

275




275







  • 1




    $begingroup$
    Given we use an appropriate testing procedure to select our regularisation parameter strength, it should not penalise any models unnecessarily. (+1)
    $endgroup$
    – usεr11852
    Mar 31 at 12:04












  • 1




    $begingroup$
    Given we use an appropriate testing procedure to select our regularisation parameter strength, it should not penalise any models unnecessarily. (+1)
    $endgroup$
    – usεr11852
    Mar 31 at 12:04







1




1




$begingroup$
Given we use an appropriate testing procedure to select our regularisation parameter strength, it should not penalise any models unnecessarily. (+1)
$endgroup$
– usεr11852
Mar 31 at 12:04




$begingroup$
Given we use an appropriate testing procedure to select our regularisation parameter strength, it should not penalise any models unnecessarily. (+1)
$endgroup$
– usεr11852
Mar 31 at 12:04










1 Answer
1






active

oldest

votes


















5












$begingroup$

For regularization terms similar to $left|thetaright|_2^2$ in effect, no they don't, they only push toward simplicity, i.e. parameters closer to zero.



Error terms such as $sum_i left|y_i - f_theta(x_i)right|_2^2$ are responsible for fighting back toward complexity (penalizing over-simplification), since the simplest model, i.e. $theta = 0$, leads to a high error.



We balance these two forces by using a regularization parameter ($lambda$) in a summation like
$$frac1Nsum_i=1^N left|y_i - f_theta(x_i)right|_2^2 + lambdaleft|thetaright|_2^2,$$
where higher $lambda$ forces the model toward more simplicity.






share|cite|improve this answer











$endgroup$












  • $begingroup$
    So, regularization like L2, L1 correspond to the first case, right?
    $endgroup$
    – alienflow
    Mar 31 at 12:05






  • 1




    $begingroup$
    @alienflow yes they all force toward zero (most simple).
    $endgroup$
    – Esmailian
    Mar 31 at 12:06











Your Answer








StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "65"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);

else
createEditor();

);

function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);



);













draft saved

draft discarded


















StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstats.stackexchange.com%2fquestions%2f400388%2fdoes-regularization-penalize-models-that-are-simpler-than-needed%23new-answer', 'question_page');

);

Post as a guest















Required, but never shown

























1 Answer
1






active

oldest

votes








1 Answer
1






active

oldest

votes









active

oldest

votes






active

oldest

votes









5












$begingroup$

For regularization terms similar to $left|thetaright|_2^2$ in effect, no they don't, they only push toward simplicity, i.e. parameters closer to zero.



Error terms such as $sum_i left|y_i - f_theta(x_i)right|_2^2$ are responsible for fighting back toward complexity (penalizing over-simplification), since the simplest model, i.e. $theta = 0$, leads to a high error.



We balance these two forces by using a regularization parameter ($lambda$) in a summation like
$$frac1Nsum_i=1^N left|y_i - f_theta(x_i)right|_2^2 + lambdaleft|thetaright|_2^2,$$
where higher $lambda$ forces the model toward more simplicity.






share|cite|improve this answer











$endgroup$












  • $begingroup$
    So, regularization like L2, L1 correspond to the first case, right?
    $endgroup$
    – alienflow
    Mar 31 at 12:05






  • 1




    $begingroup$
    @alienflow yes they all force toward zero (most simple).
    $endgroup$
    – Esmailian
    Mar 31 at 12:06















5












$begingroup$

For regularization terms similar to $left|thetaright|_2^2$ in effect, no they don't, they only push toward simplicity, i.e. parameters closer to zero.



Error terms such as $sum_i left|y_i - f_theta(x_i)right|_2^2$ are responsible for fighting back toward complexity (penalizing over-simplification), since the simplest model, i.e. $theta = 0$, leads to a high error.



We balance these two forces by using a regularization parameter ($lambda$) in a summation like
$$frac1Nsum_i=1^N left|y_i - f_theta(x_i)right|_2^2 + lambdaleft|thetaright|_2^2,$$
where higher $lambda$ forces the model toward more simplicity.






share|cite|improve this answer











$endgroup$












  • $begingroup$
    So, regularization like L2, L1 correspond to the first case, right?
    $endgroup$
    – alienflow
    Mar 31 at 12:05






  • 1




    $begingroup$
    @alienflow yes they all force toward zero (most simple).
    $endgroup$
    – Esmailian
    Mar 31 at 12:06













5












5








5





$begingroup$

For regularization terms similar to $left|thetaright|_2^2$ in effect, no they don't, they only push toward simplicity, i.e. parameters closer to zero.



Error terms such as $sum_i left|y_i - f_theta(x_i)right|_2^2$ are responsible for fighting back toward complexity (penalizing over-simplification), since the simplest model, i.e. $theta = 0$, leads to a high error.



We balance these two forces by using a regularization parameter ($lambda$) in a summation like
$$frac1Nsum_i=1^N left|y_i - f_theta(x_i)right|_2^2 + lambdaleft|thetaright|_2^2,$$
where higher $lambda$ forces the model toward more simplicity.






share|cite|improve this answer











$endgroup$



For regularization terms similar to $left|thetaright|_2^2$ in effect, no they don't, they only push toward simplicity, i.e. parameters closer to zero.



Error terms such as $sum_i left|y_i - f_theta(x_i)right|_2^2$ are responsible for fighting back toward complexity (penalizing over-simplification), since the simplest model, i.e. $theta = 0$, leads to a high error.



We balance these two forces by using a regularization parameter ($lambda$) in a summation like
$$frac1Nsum_i=1^N left|y_i - f_theta(x_i)right|_2^2 + lambdaleft|thetaright|_2^2,$$
where higher $lambda$ forces the model toward more simplicity.







share|cite|improve this answer














share|cite|improve this answer



share|cite|improve this answer








edited Mar 31 at 12:10

























answered Mar 31 at 12:00









EsmailianEsmailian

42615




42615











  • $begingroup$
    So, regularization like L2, L1 correspond to the first case, right?
    $endgroup$
    – alienflow
    Mar 31 at 12:05






  • 1




    $begingroup$
    @alienflow yes they all force toward zero (most simple).
    $endgroup$
    – Esmailian
    Mar 31 at 12:06
















  • $begingroup$
    So, regularization like L2, L1 correspond to the first case, right?
    $endgroup$
    – alienflow
    Mar 31 at 12:05






  • 1




    $begingroup$
    @alienflow yes they all force toward zero (most simple).
    $endgroup$
    – Esmailian
    Mar 31 at 12:06















$begingroup$
So, regularization like L2, L1 correspond to the first case, right?
$endgroup$
– alienflow
Mar 31 at 12:05




$begingroup$
So, regularization like L2, L1 correspond to the first case, right?
$endgroup$
– alienflow
Mar 31 at 12:05




1




1




$begingroup$
@alienflow yes they all force toward zero (most simple).
$endgroup$
– Esmailian
Mar 31 at 12:06




$begingroup$
@alienflow yes they all force toward zero (most simple).
$endgroup$
– Esmailian
Mar 31 at 12:06

















draft saved

draft discarded
















































Thanks for contributing an answer to Cross Validated!


  • Please be sure to answer the question. Provide details and share your research!

But avoid …


  • Asking for help, clarification, or responding to other answers.

  • Making statements based on opinion; back them up with references or personal experience.

Use MathJax to format equations. MathJax reference.


To learn more, see our tips on writing great answers.




draft saved


draft discarded














StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstats.stackexchange.com%2fquestions%2f400388%2fdoes-regularization-penalize-models-that-are-simpler-than-needed%23new-answer', 'question_page');

);

Post as a guest















Required, but never shown





















































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown

































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown







-

Popular posts from this blog

Quoting Keynes in a lectureIs differentiated instruction permitted by universities?How to make students learn prerequisitesUnsatisfactory Instructor Evaluations: balancing of expectations of engineering studentsWhat is the difference between a “statistician”, “applied statistician”, and an academic applying advanced stats within their field?Listing in reference section, but not quotingHow to efficiently use time while preparing for a class?Graduate Admissions: Teaching Emphasisstrategies for sharing teaching information with universities I don't personally have contacts withIs there an efficient way to give a large class of students feedback about their assignments?Is it unreasonable to expect students to read the lecture notes before attending the first class?

Image
Meta programming: Declare a new struct on the fly Is there any significance to the Valyrian Stone vault door of Qarth? Why does this part of the Space Shuttle launch pad seem to be floating in air? Make "apt-get update" show the exact output as `apt update` How did Monica know how to operate Carol's "designer"? Latex for-and in equation Hostile work environment after whistle-blowing on coworker and our boss. What do I do? Lightning Web Components - Not available in app builder Can a Bard use an arcane focus? Teaching indefinite integrals Can the electrostatic force be infinite in magnitude? Suggestion on Missing Index Creation Watchers of the Word Wall What if somebody invests in my application? What will be the benefits of Brexit? Organic chemistry Iodoform Reaction Is there a word to describe the feeling of being transfixed out of horror? You're three for three Freedom of speech and where it applies I'm in charge o...
Read more

Rank groups within a grouped sequence of TRUE/FALSE and NAGrouping functions (tapply, by, aggregate) and the *apply familyCharacters counting and subletting specific patternsWhat is the purpose of setting a key in data.table?data.table vs dplyr: can one do something well the other can't or does poorly?how to make a bar plot for a list of dataframes?How to group by unique values in a list in RPandas - Alternative to rank() function that gives unique ordinal ranks for a columnRank within group in for loop in RData transformation: from dyadic to observational data in RGetting map from purrr to work with paste0

Image
Understanding trademark infringements in a world where many dictionary words are trademarks? How to use dependency injection and avoid temporal coupling? How to write a 12-bar blues melody Can you Ready a Bard spell to release it after using Battle Magic? Why did the Apollo 13 crew extend the LM landing gear? SafeCracker #3 - We've Been Blocked As a Bard multi-classing into Warlock, what spells do I get? Multiple SQL versions with Docker Didn't attend field-specific conferences during my PhD; how much of a disadvantage is it? Copy previous line to current line from text file Would glacier 'trees' be plausible? As matter approaches a black hole, does it speed up? Find the cheapest shipping option based on item weight Python - What if the end-user didn't have the required library? Why does this derived table improve performance? Appropriate certificate to ask for a fibre installation (ANSI/TIA-568.3-D?) How can internet speed be 10 time...
Read more

Are all passive ability checks floors for active ability checks?Does passive perception supersede active perception?Which skills can be used passively?Active Opposition with Free-Form Professions in Fate5E Trap/Ambush/Stealth Mechanics VS Passive Perception ConfusionInteraction between perception and stealth in obscured conditionsHow does Keen Sight affect Passive Perception?Are all d20 rolls either attacks, saves or ability checks?Can players declare that they are making a specific ability check?Can I see a Hidden creature that is not obscured at all?Can a Stealth check ever be made passively?Is this alternate version of the Observant feat balanced?What is the minimum amount of skill points per HD?

Image
"of which" is correct here? Instead of a Universal Basic Income program, why not implement a "Universal Basic Needs" program? Bach's Toccata and Fugue in D minor breaks the "no parallel octaves" rule? Bacteria contamination inside a thermos bottle I got the following comment from a reputed math journal. What does it mean? Why do passenger jet manufacturers design their planes with stall prevention systems? A single argument pattern definition applies to multiple-argument patterns? How to pronounce "I ♥ Huckabees"? Is honey really a supersaturated solution? Does heating to un-crystalize redissolve it or melt it? Are Roman Catholic priests ever addressed as pastor How could an airship be repaired midflight? How to write cleanly even if my character uses expletive language? Describing a chess game in a novel The German vowel “a” changes to the English “i” How to make healing in an exploration game interesting Brexit...
Read more