Trjtdtk

How to give very negative feedback gracefully?

Pawn Promotion Double Checks

What are the differences between credential stuffing and password spraying?

Unknowingly ran an infinite loop in terminal

What are the spoon bit of a spoon and fork bit of a fork called?

Moving the subject of the sentence into a dangling participle

Has a commercial or military jet bi-plane ever been manufactured?

Why wasn't the Night King naked in S08E03?

My ID is expired, can I fly to the Bahamas with my passport?

Enumerate Derangements

How to get a product new from and to date in phtml file in magento 2

Accidentally deleted the "/usr/share" folder

Why do we use caret (^) as the symbol for ctrl/control?

Python password manager

Is Cola "probably the best-known" Latin word in the world? If not, which might it be?

Why is parseInt(021, 8) === 15?

Besides the up and down quark, what other quarks are present in daily matter around us?

Where can I go to avoid planes overhead?

What happens to the Time Stone?

A foe leaves the reach of my 5-foot reach sword. Can I make an Opportunity Attack with my 10-foot reach whip?

Returning the outputs of a nested structure

Summing the values of a sequence using expl3

Junior developer struggles: how to communicate with management?

Why is Arya visibly scared in the library in S8E3?

Classifier performance evaluation

Ensemble model overfitting?Balanced Linear SVM wins every class except One vs AllGradient Boosted Trees or Neural Networks Using Model Averaging?Classifier runtime evaluationClassifying Sequences Where Some Sequences in Both Classesmatching results with sklearn average_precision_scoreHow to structure data and model for multiclass classification in SVM?Class leaking on validation setBalancing XGboost still skews towards the majority classMultilabel classifcation in sklearn with soft (fuzzy) labels

1

1

$begingroup$

I have an unbalanced dataset which has 920 samples in total, 689 belong to the first class, and 222 to second class. and both classes are significant for me.
so when building a classifier model such as SVM or KNN. what measurement should I consider to evaluate the performance of the classifier? usually people use accuracy. but in my case some times I get high accuracy but zero specificity which clearly indicates that the class is biased towards the majority class (class one in my case). I've been advised to use the F-score which combines both specificity and sensitivity. Also, there is the AUC.
so what do you suggest?

classification accuracy evaluation

share|improve this question

asked Apr 10 at 12:04

gingin

1949

$endgroup$

add a comment | 

1

1

$begingroup$

I have an unbalanced dataset which has 920 samples in total, 689 belong to the first class, and 222 to second class. and both classes are significant for me.
so when building a classifier model such as SVM or KNN. what measurement should I consider to evaluate the performance of the classifier? usually people use accuracy. but in my case some times I get high accuracy but zero specificity which clearly indicates that the class is biased towards the majority class (class one in my case). I've been advised to use the F-score which combines both specificity and sensitivity. Also, there is the AUC.
so what do you suggest?

classification accuracy evaluation

share|improve this question

asked Apr 10 at 12:04

gingin

1949

$endgroup$

add a comment | 

$begingroup$

I have an unbalanced dataset which has 920 samples in total, 689 belong to the first class, and 222 to second class. and both classes are significant for me.
so when building a classifier model such as SVM or KNN. what measurement should I consider to evaluate the performance of the classifier? usually people use accuracy. but in my case some times I get high accuracy but zero specificity which clearly indicates that the class is biased towards the majority class (class one in my case). I've been advised to use the F-score which combines both specificity and sensitivity. Also, there is the AUC.
so what do you suggest?

classification accuracy evaluation

share|improve this question

asked Apr 10 at 12:04

gingin

1949

$endgroup$

share|improve this question

asked Apr 10 at 12:04

gingin

1949

share|improve this question

asked Apr 10 at 12:04

gingin

1949

asked Apr 10 at 12:04

gingin

1949

asked Apr 10 at 12:04

gingin

1949

2 Answers
2

active

oldest

votes

4

$begingroup$

Useful metrics in such scenario are:

• 
  F1 Score (and precision / recall)


• 
  ROC Curves (Metric is : Area Under the ROC Curve (AUC))

Few articles on how to choose metrics for a specific project are:

• Evaluation Metrics, ROC-Curves and imbalanced datasets by David S. Batista,




• What metrics should be used for evaluating a model on an imbalanced data set? by Shir Meir Lador,




• Choosing the Right Metric for Evaluating Machine Learning Models — Part 2 by Alvira Swalin.

share|improve this answer

edited Apr 13 at 4:04

answered Apr 10 at 12:35

Shamit VermaShamit Verma

1,6891414

$endgroup$

• 
  
  
  

  
  1
  

  
  
  
  
  
  

  
  $begingroup$
  I have something to add: The ROC curve is not a metric, the Area Under the ROC Curve (AUC) is the metric. ROC is the graphic tool to visually assess the performance of the model.
  $endgroup$
  – Juan Esteban de la Calle
  Apr 12 at 17:32
  
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  Yes, AUC is a better description of metric.
  $endgroup$
  – Shamit Verma
  Apr 13 at 3:47
  

  
  
  
  

add a comment | 

0

$begingroup$

There are many methods to measure the performance in case of data imbalance problem. I like the average per-class accuracy. You calculate the accuracy of each class and then you find the average of these classes accuracy.

share|improve this answer

answered Apr 12 at 6:31

Bashar HaddadBashar Haddad

1,2821413

$endgroup$

add a comment | 

Your Answer

StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "557"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);

else
createEditor();

);

function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);



);

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);

Sign up using Google

Sign up using Facebook

Sign up using Email and Password

Post as a guest

Name

Email

Required, but never shown

StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f49037%2fclassifier-performance-evaluation%23new-answer', 'question_page');

);

Post as a guest

Name

Email

Required, but never shown

4

$begingroup$

Useful metrics in such scenario are:

• 
  F1 Score (and precision / recall)


• 
  ROC Curves (Metric is : Area Under the ROC Curve (AUC))

Few articles on how to choose metrics for a specific project are:

• Evaluation Metrics, ROC-Curves and imbalanced datasets by David S. Batista,




• What metrics should be used for evaluating a model on an imbalanced data set? by Shir Meir Lador,




• Choosing the Right Metric for Evaluating Machine Learning Models — Part 2 by Alvira Swalin.

share|improve this answer

edited Apr 13 at 4:04

answered Apr 10 at 12:35

Shamit VermaShamit Verma

1,6891414

$endgroup$

• 
  
  
  

  
  1
  

  
  
  
  
  
  

  
  $begingroup$
  I have something to add: The ROC curve is not a metric, the Area Under the ROC Curve (AUC) is the metric. ROC is the graphic tool to visually assess the performance of the model.
  $endgroup$
  – Juan Esteban de la Calle
  Apr 12 at 17:32
  
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  Yes, AUC is a better description of metric.
  $endgroup$
  – Shamit Verma
  Apr 13 at 3:47
  

  
  
  
  

add a comment | 

4

$begingroup$

Useful metrics in such scenario are:

• 
  F1 Score (and precision / recall)


• 
  ROC Curves (Metric is : Area Under the ROC Curve (AUC))

Few articles on how to choose metrics for a specific project are:

• Evaluation Metrics, ROC-Curves and imbalanced datasets by David S. Batista,




• What metrics should be used for evaluating a model on an imbalanced data set? by Shir Meir Lador,




• Choosing the Right Metric for Evaluating Machine Learning Models — Part 2 by Alvira Swalin.

share|improve this answer

edited Apr 13 at 4:04

answered Apr 10 at 12:35

Shamit VermaShamit Verma

1,6891414

$endgroup$

• 
  
  
  

  
  1
  

  
  
  
  
  
  

  
  $begingroup$
  I have something to add: The ROC curve is not a metric, the Area Under the ROC Curve (AUC) is the metric. ROC is the graphic tool to visually assess the performance of the model.
  $endgroup$
  – Juan Esteban de la Calle
  Apr 12 at 17:32
  
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  Yes, AUC is a better description of metric.
  $endgroup$
  – Shamit Verma
  Apr 13 at 3:47
  

  
  
  
  

add a comment | 

$begingroup$

Useful metrics in such scenario are:

• 
  F1 Score (and precision / recall)


• 
  ROC Curves (Metric is : Area Under the ROC Curve (AUC))

Few articles on how to choose metrics for a specific project are:

• Evaluation Metrics, ROC-Curves and imbalanced datasets by David S. Batista,




• What metrics should be used for evaluating a model on an imbalanced data set? by Shir Meir Lador,




• Choosing the Right Metric for Evaluating Machine Learning Models — Part 2 by Alvira Swalin.

share|improve this answer

edited Apr 13 at 4:04

answered Apr 10 at 12:35

Shamit VermaShamit Verma

1,6891414

$endgroup$

share|improve this answer

edited Apr 13 at 4:04

answered Apr 10 at 12:35

Shamit VermaShamit Verma

1,6891414

answered Apr 10 at 12:35

Shamit VermaShamit Verma

1,6891414

answered Apr 10 at 12:35

Shamit VermaShamit Verma

1,6891414

0

$begingroup$

There are many methods to measure the performance in case of data imbalance problem. I like the average per-class accuracy. You calculate the accuracy of each class and then you find the average of these classes accuracy.

share|improve this answer

answered Apr 12 at 6:31

Bashar HaddadBashar Haddad

1,2821413

$endgroup$

add a comment | 

0

$begingroup$

There are many methods to measure the performance in case of data imbalance problem. I like the average per-class accuracy. You calculate the accuracy of each class and then you find the average of these classes accuracy.

share|improve this answer

answered Apr 12 at 6:31

Bashar HaddadBashar Haddad

1,2821413

$endgroup$

add a comment | 

$begingroup$

There are many methods to measure the performance in case of data imbalance problem. I like the average per-class accuracy. You calculate the accuracy of each class and then you find the average of these classes accuracy.

share|improve this answer

answered Apr 12 at 6:31

Bashar HaddadBashar Haddad

1,2821413

$endgroup$

share|improve this answer

answered Apr 12 at 6:31

Bashar HaddadBashar Haddad

1,2821413

answered Apr 12 at 6:31

Bashar HaddadBashar Haddad

1,2821413

answered Apr 12 at 6:31

Bashar HaddadBashar Haddad

1,2821413