Trjtdtk

Suppose I want to predict some event probability with a set of features. Some of the features may be gender or some other categorical variable. Assume the probabilities are well calibrated.

Now, say, I need to report the average predicted probability for males and females (and maybe some other subset of features).

Is there a difference between

1. averaging the probabilities of the full model or


2. simply fitting a model on only the subset of variables?

And if so why? The full model would outperform the simpler model by a large margin. An example would also be appreciated.

A very thought provoking question.

Surprisingly, second approach (subset) is better in theory. The first one (full) is an unbiased estimator of second one (subset). That is, expectation of average of probabilities in full model is equal to the probability in subset model.

I said "in theory", because we assume that full and subset models estimate the true P(event | full) and P(event | subset) exactly respectively (zero error, perfect generalization, etc.). In practice, choice of learning algorithm and training set affects how accurate they can fulfill this assumption. A model could be Naive Bayes, Logistic Regression, Neural Network with a SoftMax layer, etc.

Proof

Let

1. 
   $F_f$ denote the full features,


2. 
   $F_s$ denote the subset features (that we group on),


3. 
   $F_d=F_f-F_s$ denote the features that we average out,


4. 
   $boldsymbolA=F_s(x)=a$ denote the set of all possible instances with subset features equal to $a$, and


5. 
   $A subset boldsymbolA$ denote the observed subset of $boldsymbolA$ in training set.

Here is the average of probabilities in full model (first approach):
$$beginalign*
P_f(e|F_s=a)&=frac1N_Asum_xin AP(e|F_d(x),a)
endalign*$$

Expectation of above average is:
$$beginalign*
E[P_f(e|F_s=a)]&=frac1N_Asum_xin AE[P(e|F_d(x),a)]\
&=frac1N_A N_A E[P(e|F_f(X))|X in boldsymbolA]\
&=sum_bP(e|F_d=b,F_s=a)P(F_d=b|F_s=a)\
&=sum_bP(e,F_d=b|F_s=a)\
&=P(e|F_s=a)
endalign*$$

where $P(e|F_s=a)$ is the output of second approach.

Example

To verify, here is an example:

$e=texthas_job$

$F_f=textsex, texthas_degree$

$F_s=textsex$

$F_d=texthas_degree$

Full model sees:

x sex has_degree has_job
1 f 1 0
2 f 1 1
3 f 0 0

4 m 0 0
5 m 0 0
6 m 1 1

And builds

sex has_degree P(has_job=1|sex, has_degree)
f 0 0
f 1 0.5
m 0 0
m 1 1.0

And answers $P(texthas_job=1|textsex=f)$ with

$$frac13sum_x in 1, 2, 3P(texthas_job=1|x)=(0.5+0.5+0)/3=0.33,$$

and answers $P(texthas_job=1|textsex=m)$ with

$$frac13sum_x in 4, 5, 6P(texthas_job=1|x)=(0+0+1.0)/3=0.33$$

On the other hand, subset model sees:

x sex has_job
1 f 0
2 f 1
3 f 0

4 m 0
5 m 0
6 m 1

And builds

sex P(has_job=1|sex)
f 0.33
m 0.33

And answers $P(texthas_job=1|textsex=f)$ with 0.33, and $P(texthas_job=1|textsex=m)$ with 0.33.