Trjtdtk

Logistic regression, based on the logistic function $sigma(x) =
frac11 + exp(-x)$, can be seen as a hypothesis testing problem. Where the reference distribution is the standard Logistic distribution where the p.m.f is

$f(x) = fracexp(-x)[1 + exp(-x)]^2$

and the c.d.f is

$F(x) = sigma(x) = frac11 + exp(-x)$

The hypothesis to test is

$H_0: x text isn't positive hspace2.0cm H_1: x text is positive$

The test statistic is $F(x)$. We reject $H_0$ if $F(x) geq alpha$ where $alpha$ is the level of significance (in terms of hypothesis testing) or classification threshold (in terms of classification problem)

My question is that why they don't come up with the Standard normal distribution, which truly reflects the "distribution of nature", instead of Logistic distribution ?

Nice comparison.

Generally, we are allowed to experiment with as many distributions as we want, and find the one that suits our purpose. However, the normality assumption leads to an intractable derivation consisting of the notorious erf function.

Let's first pinpoint what is $x$ in the context of logistic regression. Logistic regression model can be written as:
$$P(y=1|boldsymbolx)=frac11+e^-boldsymbolw^tboldsymbolx=F(boldsymbolw^tboldsymbolx)$$
So your $x$ is actually $z=boldsymbolw^tboldsymbolx$. This means, although it is reasonable to assume that predicate $boldsymbolx$ comes from a normal distribution, the same argument does not hold for a linear combination of its dimensions, i.e. $z$. In other words, the normal assumption is not as natural for $z$ as for $boldsymbolx$.

But still, let's see what happens with normal assumption. The problem that we face here is analytical intractability. More specifically, to fit a similar model to observations using Maximum Likelihood, we need (1) derivative of cumulative distribution function (CDF) with respect to each parameter $w_i$, and (2) value of CDF for a given $z$ (see this lecture section 12.2.1 for more details).

For logistic distribution, the required gradient would be:
$$beginalign*
fracpartial F(boldsymbolx;boldsymbolw)partial w_i&=fracpartial (1+e^-boldsymbolw^tboldsymbolx)^-1partial w_i= x_i e^-boldsymbolw^tboldsymbolx(1+e^-boldsymbolw^tboldsymbolx)^-2 =x_if(boldsymbolx;boldsymbolw)
endalign*$$

However for normal distribution, CDF is the erf function which does not have an exact formula, though, its gradient is tractable. Assuming $z sim mathcalN(0, 1)$, the gradient would be:
$$beginalign*
fracpartial F(boldsymbolx;boldsymbolw)partial w_i&=fracpartial left(frac12+frac12texterfleft(fraczsqrt2right)right)partial w_i=fracx_isqrt2 pi e^-frac(boldsymbolw^tboldsymbolx)^22=x_if(boldsymbolx;boldsymbolw)
endalign*$$

In summary, the normality assumption is not as justified for $z=boldsymbolw^tboldsymbolx$ as for $boldsymbolx$, and it leads to an intractable CDF. Therefore, we continue using the good old logistic regression!

Here is a visual comparison of normal and logistic CDFs:

taken from a post by Enrique Pinzon, which implies a large analytical cost for a small difference!