Trjtdtk

I trained decision tree both in python and R, but I think the way feature importance is calculated in R may be wrong. Following is the sample code which you can use to reproduce the problem. Let's say I am predicting Income of population of 1000 based on gender and country.

x = data.frame(gender=sample(c("M","F"),n,T), country=sample(c("A","I"),n,T))
x$income = ifelse(x$gender=="M", rnorm(n, 100, 10), rnorm(n, 80, 10))
x$income = x$income + ifelse(x$country=="A", rnorm(n, 100, 10), rnorm(n, 80, 10))
write.csv(x, "data.csv")

Then lets fit a decision tree in R with max depth of 1.

fit = rpart(income~., data = x, control=rpart.control(maxdepth=1))
caret::varImp(fit)
fit

I get the following feature importance

country 0.2507630, and
gender 0.2424981

For the tree split only at country

1) root 1000 407373.4 180.5759 
 2) country=I 481 147999.6 170.0772 * 
 3) country=A 519 157219.6 190.3060 * 

When I try again with Max depth of 2, I get feature importance as

country 0.2507630, and
gender 0.8874599

For the tree split first at country as before and then at gender

1) root 1000 407373.40 180.5759 
 2) country=I 481 147999.60 170.0772 
 4) gender=F 232 40082.49 159.2805 * 
 5) gender=M 249 55676.09 180.1367 * 
 3) country=A 519 157219.60 190.3060 
 6) gender=F 248 57546.77 180.4749 * 
 7) gender=M 271 53767.73 199.3028 * 

However, if I run similar code in python

from io import StringIO
from sklearn.tree import DecisionTreeRegressor
from sklearn.tree.export import export_graphviz
from IPython.display import Image 
from sklearn import tree
import pydot
import pandas as pd

data = pd.read_csv("data.csv")

dtree=DecisionTreeRegressor(max_depth= 1)
X = data[["gender", "country" ]]
X["gender"] = X["gender"] == 'M'
X["country"] = X["country"] == 'A'
y = data[['income']]
dtree.fit(X,y)

# Export as dot file
export_graphviz(dtree, out_file='tree.dot', 
 feature_names = X.columns,filled=True, rounded=True,
 special_characters=True)

(graph,) = pydot.graph_from_dot_file('tree.dot')
graph.write_png('tree.png')
# Display in jupyter notebook
from IPython.display import Image
Image(filename = 'tree.png')

for max depth of 1 I get feature importance as

gender 0, and 
country 1 

and for max depth of two

gender 0.49, and 
country 0.51 

Now I have following two questions

1) In R, when I select max depth = 1 and split happened only at country column, then why it still gives feature importance value for gender. Even though gender is not even part of the final model. For e.g. in python it gives gender as 0 var importance.

2) Secondly, why in R, the feature importance of gender column became greater than that of country column? As the country column was more important as the initial split happened at country and not at gender. Like the values we got for python.

One of my colleague pointed out that in R, the feature importance of each column is calculated at each split. For e.g. the some feature importance value of gender and country will be calculated at first split. Then again this happens at second split. But since we already had a split based on country there will not be any information gain based on country but would be there for gender. And in the end all these importance are summed. Hence we get more value for the ones that were used at a later stage for split.