Trjtdtk

Is this Kuo-toa homebrew race balanced?

Statistical analysis applied to methods coming out of Machine Learning

Is this Half-dragon Quaggoth boss monster balanced?

Short story about astronauts fertilizing soil with their own bodies

Is a copyright notice with a non-existent name be invalid?

What was the last profitable war?

Besides transaction validation, are there any other uses of the Script language in Bitcoin

Does the transliteration of 'Dravidian' exist in Hindu scripture? Does 'Dravida' refer to a Geographical area or an ethnic group?

Vertical ranges of Column Plots in 12

New Order #6: Easter Egg

Is there a spell that can create a permanent fire?

Pointing to problems without suggesting solutions

How could a hydrazine and N2O4 cloud (or it's reactants) show up in weather radar?

"Destructive power" carried by a B-52?

Is there a verb for listening stealthily?

Weaponising the Grasp-at-a-Distance spell

One-one communication

Did any compiler fully use 80-bit floating point?

Found this skink in my tomato plant bucket. Is he trapped? Or could he leave if he wanted?

How does the body cool itself in a stillsuit?

Why is there so little support for joining EFTA in the British parliament?

Is the Mordenkainen's Sword spell underpowered?

Did John Wesley plagiarize Matthew Henry...?

The Nth Gryphon Number

Lasso implementation Drawback

1

$begingroup$

Recently I've been trying to implement Lasso by myself in R, not using the "glmnet" package, and based on an article by Tibshirani I wrote a raw code to implement coordinate descent method, and it does successfully force many coefficients to shrink to zero. But when I try to compare my result based on a famous R built-in data set(mtcars) with glmnet package, and apply different lambdas, only to find out that while other coefficients are reasonable, one coefficient that should be insignificant remain quite large. I've been trying to figure out the reason. Specifically I have some detailed questions:

1. How to deal with the intercept in the formula? When the data set is scaled, it seems that the $beta_0$ can just be set to the mean value of the response variable. But even the glmnet package produces a different intercept when lambda is changed in the model.




2. What could be the possible reason for the problem as I described above?

Hope someone could lend a hand, thanks! If any more details are needed, I can provide them.

Here's my source code in Git. I've also described my problem in detailed in README.md.

machine-learning

share|improve this question

edited Apr 6 at 11:41

DiaryofNewton

asked Apr 3 at 17:06

DiaryofNewtonDiaryofNewton

62

$endgroup$

• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  There is no need to normalise the intercept term. There are multifarious reasons, one of them is that it is just a single coefficient, but there are many other than that.
  $endgroup$
  – Vaalizaadeh
  Apr 3 at 18:01
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  Are you trying to implement just elastic net, i.e. regularized linear regression?
  $endgroup$
  – Matthew Drury
  Apr 3 at 20:25
  

  
  
  
  


• 
  
  
  

  
  1
  

  
  
  
  
  
  

  
  $begingroup$
  interesting question but it is difficult to answer since it could be just a bug in your code or something completely different. one thing to consider is that the algorithm is usually described for the case where the data already has mean zero and unit variance. you need to adjust the formulas accordingly if that's not true for mtcars
  $endgroup$
  – oW_♦
  Apr 3 at 23:24
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  @oW_ But what's the case for glmnet? Seems it didn't adjust the final coefficient accordingly, because many of them remain pretty close to mine, and if they get adjusted, they should become very small, since the original corresponding data are quite large
  $endgroup$
  – DiaryofNewton
  Apr 4 at 2:01
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  @MatthewDrury Elastic net. The penalty term is a mixture of l1 norm and l2 norm. What can this suggest?
  $endgroup$
  – DiaryofNewton
  Apr 4 at 2:02
  

  
  
  
  

 | 
show 5 more comments

1

$begingroup$

Recently I've been trying to implement Lasso by myself in R, not using the "glmnet" package, and based on an article by Tibshirani I wrote a raw code to implement coordinate descent method, and it does successfully force many coefficients to shrink to zero. But when I try to compare my result based on a famous R built-in data set(mtcars) with glmnet package, and apply different lambdas, only to find out that while other coefficients are reasonable, one coefficient that should be insignificant remain quite large. I've been trying to figure out the reason. Specifically I have some detailed questions:

1. How to deal with the intercept in the formula? When the data set is scaled, it seems that the $beta_0$ can just be set to the mean value of the response variable. But even the glmnet package produces a different intercept when lambda is changed in the model.




2. What could be the possible reason for the problem as I described above?

Hope someone could lend a hand, thanks! If any more details are needed, I can provide them.

Here's my source code in Git. I've also described my problem in detailed in README.md.

machine-learning

share|improve this question

edited Apr 6 at 11:41

DiaryofNewton

asked Apr 3 at 17:06

DiaryofNewtonDiaryofNewton

62

$endgroup$

• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  There is no need to normalise the intercept term. There are multifarious reasons, one of them is that it is just a single coefficient, but there are many other than that.
  $endgroup$
  – Vaalizaadeh
  Apr 3 at 18:01
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  Are you trying to implement just elastic net, i.e. regularized linear regression?
  $endgroup$
  – Matthew Drury
  Apr 3 at 20:25
  

  
  
  
  


• 
  
  
  

  
  1
  

  
  
  
  
  
  

  
  $begingroup$
  interesting question but it is difficult to answer since it could be just a bug in your code or something completely different. one thing to consider is that the algorithm is usually described for the case where the data already has mean zero and unit variance. you need to adjust the formulas accordingly if that's not true for mtcars
  $endgroup$
  – oW_♦
  Apr 3 at 23:24
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  @oW_ But what's the case for glmnet? Seems it didn't adjust the final coefficient accordingly, because many of them remain pretty close to mine, and if they get adjusted, they should become very small, since the original corresponding data are quite large
  $endgroup$
  – DiaryofNewton
  Apr 4 at 2:01
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  @MatthewDrury Elastic net. The penalty term is a mixture of l1 norm and l2 norm. What can this suggest?
  $endgroup$
  – DiaryofNewton
  Apr 4 at 2:02
  

  
  
  
  

 | 
show 5 more comments

$begingroup$

Recently I've been trying to implement Lasso by myself in R, not using the "glmnet" package, and based on an article by Tibshirani I wrote a raw code to implement coordinate descent method, and it does successfully force many coefficients to shrink to zero. But when I try to compare my result based on a famous R built-in data set(mtcars) with glmnet package, and apply different lambdas, only to find out that while other coefficients are reasonable, one coefficient that should be insignificant remain quite large. I've been trying to figure out the reason. Specifically I have some detailed questions:

1. How to deal with the intercept in the formula? When the data set is scaled, it seems that the $beta_0$ can just be set to the mean value of the response variable. But even the glmnet package produces a different intercept when lambda is changed in the model.




2. What could be the possible reason for the problem as I described above?

Hope someone could lend a hand, thanks! If any more details are needed, I can provide them.

Here's my source code in Git. I've also described my problem in detailed in README.md.

machine-learning

share|improve this question

edited Apr 6 at 11:41

DiaryofNewton

asked Apr 3 at 17:06

DiaryofNewtonDiaryofNewton

62

$endgroup$

share|improve this question

edited Apr 6 at 11:41

DiaryofNewton

asked Apr 3 at 17:06

DiaryofNewtonDiaryofNewton

62

share|improve this question

edited Apr 6 at 11:41

DiaryofNewton

asked Apr 3 at 17:06

DiaryofNewtonDiaryofNewton

62

asked Apr 3 at 17:06

DiaryofNewtonDiaryofNewton

62

asked Apr 3 at 17:06

DiaryofNewtonDiaryofNewton

62