Trjtdtk

I have two questions :

1. Why doesn't normalization have any effect on linear regressor performance (mathematical approach is appreciated ) ?


2. When we normalize the training set we ought to normalize the target set too . Won't it affect the performance ? I mean won't the data set change completely because the model had different Ranges of features as compared to the ranges of features in target set .

I tried googling the questions but was not able to come to conclusion . Any help would be appreciated .

Thanks !

1. Why doesn't normalization have any effect on linear regressor performance (mathematical approach is appreciated)?

Theoretically, normalization does not influence the performance of the model. In order to understand this let us have a look at a standard linear regression.
$$y_i = boldsymbolw^Tboldsymbolx_i + b+varepsilon_i$$

If we Introduce the scaled independent variable $boldsymbolz_i=dfrac1sigmaleft[ boldsymbolx_i - barboldsymbolxright] implies boldsymbolx_i=barboldsymbolx+sigmaboldsymbolz_i$. This will result in
$$y_i = boldsymbolw^Tbarboldsymbolx+sigmaboldsymbolw^Tboldsymbolz_i+b+varepsilon_i.$$

If we introduce $tildeb=b+boldsymbolw^Tbarboldsymbolx$ and $tildeboldsymbolw^T=sigmaboldsymbolw^T$ we can rewrite the equation into

$$y_i=tildeboldsymbolw^Tboldsymbolz_i+tildeb+varepsilon_i.$$

Hence, we see that the transformed independent variables just change the bias (if a translation is included) and the weights are scaled by $sigma$. The significance of the parameters will not change only their specified values.

But if normalization does not really enhance the model, why do we still do it? The reason is more computationally inspired. If we had very large values the weights would need to be very small such that we can scale the output of the regression into a reasonable range. A large range of numerical values forces us to reserve more memory for our variables that we use. Hence, it is better to normalize our inputs, such that the parameters don't need to scale the inputs down to a reasonable amount that it fits the output.

2. When we normalize the training set we ought to normalize the target set too. Won't it affect the performance? I mean won't the data set
   change completely because the model had different ranges of features
   as compared to the ranges of features in the target set.

Normalizing the output is not necessary, but it can also improve the numerical efficiency. You can just use the previous linear transformation on your dependent variable (output) and you will see that you can rewrite it to a standard linear regression in the new output. Just remember to transform your inputs and retransform your outputs if you want to use the original variables.

Does the significance of the parameters change?

In order to show that scaling the inputs by a constant factor $sigma$ does not influence the significance of the parameters, we will calculate the $t$-value for a given coefficient. If the $t$-value stays invariant the $p$-value will also stay invariant.

For the linear regression $y_i = boldsymbolwboldsymbolx_i+varepsilon_i$ (bias absorbed into the weight vector). The regression coefficients are given by

$$hatboldsymbolw=left[boldsymbolX^TboldsymbolX right]^-1boldsymbolX^Tboldsymbolyquad text , in which quad
boldsymbolX=beginbmatrixboldsymbolx_1^T \ vdots \ boldsymbolx_N^Tendbmatrix$$

is the data matrix with an added $1$-column for the bias.

Additionally, we need the matrix
$$boldsymbolC=left[boldsymbolX^TboldsymbolX right]^-1quad text and quad s_e = sqrtdfrac(boldsymboly-hatboldsymboly)^T(boldsymboly-hatboldsymboly)N-p-1,$$

in which $N$ is the number of observations, $p$ is the number of predictors, $boldsymboly$ is the vector of outputs and $hatboldsymboly$ is the vector of predicted outputs. We saw that the predicted values will stay invariant under scaling. Hence $s_e$ is invariant under scaling.

The $t$-value for a regression weight given by

$$t_i= dfrachatw_i-mathbbEleft[hatw_iright]s_esqrtc_ii.$$

The $c_ii$-values are the corresponding diagonal values of the $boldsymbolC$ matrix.

If we scale our inputs by $sigma$ (we ignore the 1 column of the data set which is only important for the bias) then we observe

$$boldsymbolX'=sigmaboldsymbolX
text , hatboldsymbolw' = dfrac1sigmahatboldsymbolw text , mathbbEleft[ hatboldsymbolw'right]=dfrac1sigmamathbbEleft[ hatboldsymbolwright] quad text and quad boldsymbolC' = dfrac1sigma^2boldsymbolC.$$
The last condition implies
$$implies c_ii' = dfrac1sigma^2c_ii implies sqrtc_ii' = dfrac1sigmasqrtc_ii.$$

By these observations, we see that the $t$-value stays invariant.

$$t' = dfrachatw_i'-mathbbEleft[hatw_i' right]s_esqrtc_ii'=dfrachatw_i-mathbbEleft[hatw_i right]s_esqrtc_ii=t$$

Hence, the significance of the regression coefficients didn't change either. In theory, the same should apply individually scaling the variables but the algebra gets more complicated.