Trjtdtk

In my use-case of multi-class classification, my data distribution is like below:

It might be too silly to ask this (and possibly could be gravely wrong), but is there a provision to default an o/p class to a value which is safe to be defaulted than to be predicted a completely wrong outcome.

Ex. Suppose the case, where an incoming email meant to be for "hardware" department but is predicted as for "Company Leadership" department and hence routed to all senior members of the company. In such case, since the prediction accuracy of the entire output class is below say 20% accuracy, I would like to default that entire class to "service desk" group and let them manually sort it.

Hope I made my question clear (Might be confusing as well!). Please let me know if any clarifications required. I would be happy to amend the wordings.

Thanks. :)

I don't think there is a standard method to do this. But if you use a probabilistic model you can use the predicted probability together with thresholds on each class to only allow classifications that you deem certain enough. Then if the class with the highest probability does not meet the threshold you can set it to the default class.

I tested it out with sklearn and numpy, this could be an approach:

# Train probabilistic classifier
clf.fit(X_train, y_train)

# Get probabilities
probas = clf.predict_proba(X_test)

# Get the class with highest probability
highest_proba_class = np.argmax(probas, axis=1)

# Set different thresholds per class
thresholds = np.array([0.9, 0.2, 0.5])

# Init our prediction array
predictions = np.zeros_like(highest_proba_class)

# Set a default class to set if we don't reach threshold
default_class = 2

# Loop over predictions
for idx, highest_class in enumerate(highest_proba_class):

 # Threshold check if threshold was met, otherwise set default
 if probas[idx][highest_class] >= thresholds[highest_class]:
 predictions[idx] = highest_class
 else:
 predictions[idx] = default_class

There is a interesting module in scikitlearn in python which can help you a lot for what you are trying to do:

Dummy Classifiers in scikitlearn

Is very easy to use and you can select different methods.