Doubt to use accuracy or macro f1 measure in an unbalanced classification taskOver-fitting issue in a classification problem (unbalanced data)Which accuracy metric of a ML classifier can maximize map@K of a recommender system for an unbalanced dataset?Balanced Linear SVM wins every class except One vs Allunbalanced data classificationWhy MLP only learns bias for unbalanced binary classification?Hyperparameter tuning in multiclass classification problem: which scoring metric?How to deal with unbalanced data in pixelwise classification?multi class classification : unbalanced data - good testing results poor prediction resultsAudio classification data balanceMicro-F1 and Macro-F1 are equal in binary classification and I don't know why
Matrix using tikz package
What does "^L" mean in C?
gerund and noun applications
Pronounciation of the combination "st" in spanish accents
Differential and Linear trail propagation in Noekeon
Calculate the frequency of characters in a string
Why is there so much iron?
If "dar" means "to give", what does "daros" mean?
How do hiring committees for research positions view getting "scooped"?
Maths symbols and unicode-math input inside siunitx commands
Worshiping one God at a time?
How does one measure the Fourier components of a signal?
The average age of first marriage in Russia
What (if any) is the reason to buy in small local stores?
Comment Box for Substitution Method of Integrals
Does multi-classing into Fighter give you heavy armor proficiency?
Print last inputted byte
Suggestions on how to spend Shaabath (constructively) alone
Can other pieces capture a threatening piece and prevent a checkmate?
How to define limit operations in general topological spaces? Are nets able to do this?
How can an organ that provides biological immortality be unable to regenerate?
Violin - Can double stops be played when the strings are not next to each other?
Is it insecure to send a password in a `curl` command?
Print a physical multiplication table
Doubt to use accuracy or macro f1 measure in an unbalanced classification task
Over-fitting issue in a classification problem (unbalanced data)Which accuracy metric of a ML classifier can maximize map@K of a recommender system for an unbalanced dataset?Balanced Linear SVM wins every class except One vs Allunbalanced data classificationWhy MLP only learns bias for unbalanced binary classification?Hyperparameter tuning in multiclass classification problem: which scoring metric?How to deal with unbalanced data in pixelwise classification?multi class classification : unbalanced data - good testing results poor prediction resultsAudio classification data balanceMicro-F1 and Macro-F1 are equal in binary classification and I don't know why
$begingroup$
I have a multi-class classification task where the organizers said that the final results will be using the Accuracy measure.
The provided data is unbalanced, and I don't have an idea about the test set (is it balanced or not), but I think it will be balanced since they use accuracy.
Anyway ..
My question: Is it a good idea to tune my system using F1-macro rather than Accuracy? since the training data is unbalanced.
or it's better to use the Accuracy?
classification unbalanced-classes evaluation
edited 2 days ago
Alireza Zolanvari
19114
asked Dec 5 '18 at 14:36
GhanemGhanem
1186
$endgroup$
add a comment |
$begingroup$
I have a multi-class classification task where the organizers said that the final results will be using the Accuracy measure.
The provided data is unbalanced, and I don't have an idea about the test set (is it balanced or not), but I think it will be balanced since they use accuracy.
Anyway ..
My question: Is it a good idea to tune my system using F1-macro rather than Accuracy? since the training data is unbalanced.
or it's better to use the Accuracy?
classification unbalanced-classes evaluation
edited 2 days ago
Alireza Zolanvari
19114
asked Dec 5 '18 at 14:36
GhanemGhanem
1186
$endgroup$
$begingroup$
It's a better approach to use F1. Will the organizers evaluate all your process or only your results? I'd tune my model with F1 and then deliver it with the accuracy
$endgroup$
– ignatius
Dec 5 '18 at 15:48
1
$begingroup$
Also, they might want to evaluate how you approach the problem... The reason behind giving an unbalanced data-set and accuracy as metric might be to check whether you notice the problems with that and how you face it, for example balancing your data in some way
$endgroup$
– ignatius
Dec 5 '18 at 15:50
$begingroup$
only my results .. and thanx for the suggestion
$endgroup$
– Ghanem
Dec 5 '18 at 16:18
1
$begingroup$
Well, so nothing prevents you from tuning the model with a metric of your choice. Good luck!
$endgroup$
– ignatius
Dec 5 '18 at 16:20
add a comment |
$begingroup$
I have a multi-class classification task where the organizers said that the final results will be using the Accuracy measure.
The provided data is unbalanced, and I don't have an idea about the test set (is it balanced or not), but I think it will be balanced since they use accuracy.
Anyway ..
My question: Is it a good idea to tune my system using F1-macro rather than Accuracy? since the training data is unbalanced.
or it's better to use the Accuracy?
classification unbalanced-classes evaluation
edited 2 days ago
Alireza Zolanvari
19114
asked Dec 5 '18 at 14:36
GhanemGhanem
1186
$endgroup$
I have a multi-class classification task where the organizers said that the final results will be using the Accuracy measure.
The provided data is unbalanced, and I don't have an idea about the test set (is it balanced or not), but I think it will be balanced since they use accuracy.
Anyway ..
My question: Is it a good idea to tune my system using F1-macro rather than Accuracy? since the training data is unbalanced.
or it's better to use the Accuracy?
classification unbalanced-classes evaluation
classification unbalanced-classes evaluation
edited 2 days ago
Alireza Zolanvari
19114
asked Dec 5 '18 at 14:36
GhanemGhanem
1186
edited 2 days ago
Alireza Zolanvari
19114
asked Dec 5 '18 at 14:36
GhanemGhanem
1186
edited 2 days ago
Alireza Zolanvari
19114
edited 2 days ago
Alireza Zolanvari
19114
edited 2 days ago
Alireza Zolanvari
19114
19114
asked Dec 5 '18 at 14:36
GhanemGhanem
1186
asked Dec 5 '18 at 14:36
GhanemGhanem
1186
asked Dec 5 '18 at 14:36
GhanemGhanem
1186
1186
$begingroup$
It's a better approach to use F1. Will the organizers evaluate all your process or only your results? I'd tune my model with F1 and then deliver it with the accuracy
$endgroup$
– ignatius
Dec 5 '18 at 15:48
1
$begingroup$
Also, they might want to evaluate how you approach the problem... The reason behind giving an unbalanced data-set and accuracy as metric might be to check whether you notice the problems with that and how you face it, for example balancing your data in some way
$endgroup$
– ignatius
Dec 5 '18 at 15:50
$begingroup$
only my results .. and thanx for the suggestion
$endgroup$
– Ghanem
Dec 5 '18 at 16:18
1
$begingroup$
Well, so nothing prevents you from tuning the model with a metric of your choice. Good luck!
$endgroup$
– ignatius
Dec 5 '18 at 16:20
add a comment |
$begingroup$
It's a better approach to use F1. Will the organizers evaluate all your process or only your results? I'd tune my model with F1 and then deliver it with the accuracy
$endgroup$
– ignatius
Dec 5 '18 at 15:48
1
$begingroup$
Also, they might want to evaluate how you approach the problem... The reason behind giving an unbalanced data-set and accuracy as metric might be to check whether you notice the problems with that and how you face it, for example balancing your data in some way
$endgroup$
– ignatius
Dec 5 '18 at 15:50
$begingroup$
only my results .. and thanx for the suggestion
$endgroup$
– Ghanem
Dec 5 '18 at 16:18
1
$begingroup$
Well, so nothing prevents you from tuning the model with a metric of your choice. Good luck!
$endgroup$
– ignatius
Dec 5 '18 at 16:20
$begingroup$
It's a better approach to use F1. Will the organizers evaluate all your process or only your results? I'd tune my model with F1 and then deliver it with the accuracy
$endgroup$
– ignatius
Dec 5 '18 at 15:48
$begingroup$
It's a better approach to use F1. Will the organizers evaluate all your process or only your results? I'd tune my model with F1 and then deliver it with the accuracy
$endgroup$
– ignatius
Dec 5 '18 at 15:48
1
1
$begingroup$
Also, they might want to evaluate how you approach the problem... The reason behind giving an unbalanced data-set and accuracy as metric might be to check whether you notice the problems with that and how you face it, for example balancing your data in some way
$endgroup$
– ignatius
Dec 5 '18 at 15:50
$begingroup$
Also, they might want to evaluate how you approach the problem... The reason behind giving an unbalanced data-set and accuracy as metric might be to check whether you notice the problems with that and how you face it, for example balancing your data in some way
$endgroup$
– ignatius
Dec 5 '18 at 15:50
$begingroup$
only my results .. and thanx for the suggestion
$endgroup$
– Ghanem
Dec 5 '18 at 16:18
$begingroup$
only my results .. and thanx for the suggestion
$endgroup$
– Ghanem
Dec 5 '18 at 16:18
1
1
$begingroup$
Well, so nothing prevents you from tuning the model with a metric of your choice. Good luck!
$endgroup$
– ignatius
Dec 5 '18 at 16:20
$begingroup$
Well, so nothing prevents you from tuning the model with a metric of your choice. Good luck!
$endgroup$
– ignatius
Dec 5 '18 at 16:20
add a comment |
2 Answers
2
active
oldest
votes
$begingroup$
using accuracy for unbalance data means that correct classification for the most populous class members is more important than others. If the importance of correct classifying for all data records is equal in your problem accuracy is one of the worst choices.
There are some other good choices beside F1-macro which can be more helpful. Some of these metrics are as follows:
"Kappa", "SOA1(Landis & Koch)", "SOA2(Fleiss)", "SOA3(Altman)", "SOA4(Cicchetti)", "CEN",
"MCEN", "MCC", "J", "Overall J", "Overall MCC", "Overall CEN", "Overall MCEN", "AUC",
"AUCI", "G", "DP", "DPI", "GI"
Disclaimer:
If you use python, PyCM module can help you to find out these metrics.
Here is a simple code to get the recommended parameters from this module:
>>> from pycm import *
>>> cm = ConfusionMatrix(matrix="Class1": "Class1": 1, "Class2":2, "Class2": "Class1": 0, "Class2": 5)
>>> print(cm.recommended_list)
["Kappa", "SOA1(Landis & Koch)", "SOA2(Fleiss)", "SOA3(Altman)", "SOA4(Cicchetti)", "CEN", "MCEN", "MCC", "J", "Overall J", "Overall MCC", "Overall CEN", "Overall MCEN", "AUC", "AUCI", "G", "DP", "DPI", "GI"]
After that, each of these parameters you want to use as the loss function can be used as follows:
>>> y_pred = model.predict #the prediction of the implemented model
>>> y_actu = data.target #data labels
>>> cm = ConfusionMatrix(y_actu, y_pred)
>>> loss = cm.Kappa #or any other parameter (Example: cm.SOA1)
answered Mar 11 at 7:37
Alireza ZolanvariAlireza Zolanvari
19114
$endgroup$
add a comment |
$begingroup$
You should definitely use macro-average F1 as the accuracy could be highly biased by the majority class. The F1 makes an harmonic mean of recall and precision, giving a trade-off measure considering what has been correctly predicted and what not.
answered Mar 11 at 8:23
3nomis3nomis
1929
$endgroup$
add a comment |
Your Answer
StackExchange.ifUsing("editor", function ()
return StackExchange.using("mathjaxEditing", function ()
StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix)
StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["$", "$"], ["\\(","\\)"]]);
);
);
, "mathjax-editing");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "557"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f42177%2fdoubt-to-use-accuracy-or-macro-f1-measure-in-an-unbalanced-classification-task%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
2 Answers
2
active
oldest
votes
2 Answers
2
active
oldest
votes
active
oldest
votes
active
oldest
votes
$begingroup$
using accuracy for unbalance data means that correct classification for the most populous class members is more important than others. If the importance of correct classifying for all data records is equal in your problem accuracy is one of the worst choices.
There are some other good choices beside F1-macro which can be more helpful. Some of these metrics are as follows:
"Kappa", "SOA1(Landis & Koch)", "SOA2(Fleiss)", "SOA3(Altman)", "SOA4(Cicchetti)", "CEN",
"MCEN", "MCC", "J", "Overall J", "Overall MCC", "Overall CEN", "Overall MCEN", "AUC",
"AUCI", "G", "DP", "DPI", "GI"
Disclaimer:
If you use python, PyCM module can help you to find out these metrics.
Here is a simple code to get the recommended parameters from this module:
>>> from pycm import *
>>> cm = ConfusionMatrix(matrix="Class1": "Class1": 1, "Class2":2, "Class2": "Class1": 0, "Class2": 5)
>>> print(cm.recommended_list)
["Kappa", "SOA1(Landis & Koch)", "SOA2(Fleiss)", "SOA3(Altman)", "SOA4(Cicchetti)", "CEN", "MCEN", "MCC", "J", "Overall J", "Overall MCC", "Overall CEN", "Overall MCEN", "AUC", "AUCI", "G", "DP", "DPI", "GI"]
After that, each of these parameters you want to use as the loss function can be used as follows:
>>> y_pred = model.predict #the prediction of the implemented model
>>> y_actu = data.target #data labels
>>> cm = ConfusionMatrix(y_actu, y_pred)
>>> loss = cm.Kappa #or any other parameter (Example: cm.SOA1)
answered Mar 11 at 7:37
Alireza ZolanvariAlireza Zolanvari
19114
$endgroup$
add a comment |
$begingroup$
using accuracy for unbalance data means that correct classification for the most populous class members is more important than others. If the importance of correct classifying for all data records is equal in your problem accuracy is one of the worst choices.
There are some other good choices beside F1-macro which can be more helpful. Some of these metrics are as follows:
"Kappa", "SOA1(Landis & Koch)", "SOA2(Fleiss)", "SOA3(Altman)", "SOA4(Cicchetti)", "CEN",
"MCEN", "MCC", "J", "Overall J", "Overall MCC", "Overall CEN", "Overall MCEN", "AUC",
"AUCI", "G", "DP", "DPI", "GI"
Disclaimer:
If you use python, PyCM module can help you to find out these metrics.
Here is a simple code to get the recommended parameters from this module:
>>> from pycm import *
>>> cm = ConfusionMatrix(matrix="Class1": "Class1": 1, "Class2":2, "Class2": "Class1": 0, "Class2": 5)
>>> print(cm.recommended_list)
["Kappa", "SOA1(Landis & Koch)", "SOA2(Fleiss)", "SOA3(Altman)", "SOA4(Cicchetti)", "CEN", "MCEN", "MCC", "J", "Overall J", "Overall MCC", "Overall CEN", "Overall MCEN", "AUC", "AUCI", "G", "DP", "DPI", "GI"]
After that, each of these parameters you want to use as the loss function can be used as follows:
>>> y_pred = model.predict #the prediction of the implemented model
>>> y_actu = data.target #data labels
>>> cm = ConfusionMatrix(y_actu, y_pred)
>>> loss = cm.Kappa #or any other parameter (Example: cm.SOA1)
answered Mar 11 at 7:37
Alireza ZolanvariAlireza Zolanvari
19114
$endgroup$
add a comment |
$begingroup$
using accuracy for unbalance data means that correct classification for the most populous class members is more important than others. If the importance of correct classifying for all data records is equal in your problem accuracy is one of the worst choices.
There are some other good choices beside F1-macro which can be more helpful. Some of these metrics are as follows:
"Kappa", "SOA1(Landis & Koch)", "SOA2(Fleiss)", "SOA3(Altman)", "SOA4(Cicchetti)", "CEN",
"MCEN", "MCC", "J", "Overall J", "Overall MCC", "Overall CEN", "Overall MCEN", "AUC",
"AUCI", "G", "DP", "DPI", "GI"
Disclaimer:
If you use python, PyCM module can help you to find out these metrics.
Here is a simple code to get the recommended parameters from this module:
>>> from pycm import *
>>> cm = ConfusionMatrix(matrix="Class1": "Class1": 1, "Class2":2, "Class2": "Class1": 0, "Class2": 5)
>>> print(cm.recommended_list)
["Kappa", "SOA1(Landis & Koch)", "SOA2(Fleiss)", "SOA3(Altman)", "SOA4(Cicchetti)", "CEN", "MCEN", "MCC", "J", "Overall J", "Overall MCC", "Overall CEN", "Overall MCEN", "AUC", "AUCI", "G", "DP", "DPI", "GI"]
After that, each of these parameters you want to use as the loss function can be used as follows:
>>> y_pred = model.predict #the prediction of the implemented model
>>> y_actu = data.target #data labels
>>> cm = ConfusionMatrix(y_actu, y_pred)
>>> loss = cm.Kappa #or any other parameter (Example: cm.SOA1)
answered Mar 11 at 7:37
Alireza ZolanvariAlireza Zolanvari
19114
$endgroup$
using accuracy for unbalance data means that correct classification for the most populous class members is more important than others. If the importance of correct classifying for all data records is equal in your problem accuracy is one of the worst choices.
There are some other good choices beside F1-macro which can be more helpful. Some of these metrics are as follows:
"Kappa", "SOA1(Landis & Koch)", "SOA2(Fleiss)", "SOA3(Altman)", "SOA4(Cicchetti)", "CEN",
"MCEN", "MCC", "J", "Overall J", "Overall MCC", "Overall CEN", "Overall MCEN", "AUC",
"AUCI", "G", "DP", "DPI", "GI"
Disclaimer:
If you use python, PyCM module can help you to find out these metrics.
Here is a simple code to get the recommended parameters from this module:
>>> from pycm import *
>>> cm = ConfusionMatrix(matrix="Class1": "Class1": 1, "Class2":2, "Class2": "Class1": 0, "Class2": 5)
>>> print(cm.recommended_list)
["Kappa", "SOA1(Landis & Koch)", "SOA2(Fleiss)", "SOA3(Altman)", "SOA4(Cicchetti)", "CEN", "MCEN", "MCC", "J", "Overall J", "Overall MCC", "Overall CEN", "Overall MCEN", "AUC", "AUCI", "G", "DP", "DPI", "GI"]
After that, each of these parameters you want to use as the loss function can be used as follows:
>>> y_pred = model.predict #the prediction of the implemented model
>>> y_actu = data.target #data labels
>>> cm = ConfusionMatrix(y_actu, y_pred)
>>> loss = cm.Kappa #or any other parameter (Example: cm.SOA1)
answered Mar 11 at 7:37
Alireza ZolanvariAlireza Zolanvari
19114
answered Mar 11 at 7:37
Alireza ZolanvariAlireza Zolanvari
19114
answered Mar 11 at 7:37
Alireza ZolanvariAlireza Zolanvari
19114
answered Mar 11 at 7:37
Alireza ZolanvariAlireza Zolanvari
19114
19114
add a comment |
add a comment |
$begingroup$
You should definitely use macro-average F1 as the accuracy could be highly biased by the majority class. The F1 makes an harmonic mean of recall and precision, giving a trade-off measure considering what has been correctly predicted and what not.
answered Mar 11 at 8:23
3nomis3nomis
1929
$endgroup$
add a comment |
$begingroup$
You should definitely use macro-average F1 as the accuracy could be highly biased by the majority class. The F1 makes an harmonic mean of recall and precision, giving a trade-off measure considering what has been correctly predicted and what not.
answered Mar 11 at 8:23
3nomis3nomis
1929
$endgroup$
add a comment |
$begingroup$
You should definitely use macro-average F1 as the accuracy could be highly biased by the majority class. The F1 makes an harmonic mean of recall and precision, giving a trade-off measure considering what has been correctly predicted and what not.
answered Mar 11 at 8:23
3nomis3nomis
1929
$endgroup$
You should definitely use macro-average F1 as the accuracy could be highly biased by the majority class. The F1 makes an harmonic mean of recall and precision, giving a trade-off measure considering what has been correctly predicted and what not.
answered Mar 11 at 8:23
3nomis3nomis
1929
answered Mar 11 at 8:23
3nomis3nomis
1929
answered Mar 11 at 8:23
3nomis3nomis
1929
answered Mar 11 at 8:23
3nomis3nomis
1929
1929
add a comment |
add a comment |
Thanks for contributing an answer to Data Science Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
Use MathJax to format equations. MathJax reference.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f42177%2fdoubt-to-use-accuracy-or-macro-f1-measure-in-an-unbalanced-classification-task%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
$begingroup$
It's a better approach to use F1. Will the organizers evaluate all your process or only your results? I'd tune my model with F1 and then deliver it with the accuracy
$endgroup$
– ignatius
Dec 5 '18 at 15:48
1
$begingroup$
Also, they might want to evaluate how you approach the problem... The reason behind giving an unbalanced data-set and accuracy as metric might be to check whether you notice the problems with that and how you face it, for example balancing your data in some way
$endgroup$
– ignatius
Dec 5 '18 at 15:50
$begingroup$
only my results .. and thanx for the suggestion
$endgroup$
– Ghanem
Dec 5 '18 at 16:18
1
$begingroup$
Well, so nothing prevents you from tuning the model with a metric of your choice. Good luck!
$endgroup$
– ignatius
Dec 5 '18 at 16:20