Using an unsupervised Isolation Forest, how does one identify the optimal number of outliers from the anomaly scores?ML algorithms for defining NORMAL user behaviorHow to set the Reconstruction error threshold for anomaly detection using autoencoders?How to compare two unsupervised anomaly detection algorithms on the same data-set?
Is Cola "probably the best-known" Latin word in the world? If not, which might it be?
How did Arya get her dagger back from Sansa?
Python password manager
What is Shri Venkateshwara Mangalasasana stotram recited for?
Timing of New Changes in Change Sets
What does a yield inside a yield do?
I drew a randomly colored grid of points with tikz, how do I force it to remember the first grid from then on?
How is the law in a case of multiple edim zomemim justified by Chachomim?
Unknowingly ran an infinite loop in terminal
Point of the the Dothraki's attack in GoT S8E3?
Why do money exchangers give different rates to different bills?
When boost::lexical_cast to std::string fails?
Why Isn’t SQL More Refactorable?
Have I damaged my car by attempting to reverse with hand/park brake up?
Why isn't nylon as strong as kevlar?
Selecting a secure PIN for building access
What are the spoon bit of a spoon and fork bit of a fork called?
Airbnb - host wants to reduce rooms, can we get refund?
I need a disease
Is there formal test of non-linearity in linear regression?
What is the most remote airport from the center of the city it supposedly serves?
CRT Oscilloscope - part of the plot is missing
Why do we use caret (^) as the symbol for ctrl/control?
Ubuntu 19.04 python 3.6 is not working
Using an unsupervised Isolation Forest, how does one identify the optimal number of outliers from the anomaly scores?
ML algorithms for defining NORMAL user behaviorHow to set the Reconstruction error threshold for anomaly detection using autoencoders?How to compare two unsupervised anomaly detection algorithms on the same data-set?
$begingroup$
I am using an unsupervised isolation forest algorithm and computing anomaly scores to detect outliers from a 2 dimensional toy dataset. From a scatter plot, I am able to detect/visualize the data points with the highest anomaly scores (example: top 10 or top 15 outliers from the data)as my outliers. Is the number of outliers subjective to user decision (for example:Anomaly score of 0.5>S>1 is an outlier and everything less than 0.5 is not) or is there a way to detect the optimal number of outliers based on the anomaly scores?
unsupervised-learning data-science-model
asked Apr 9 at 18:34
user71207user71207
112
$endgroup$
add a comment |
$begingroup$
I am using an unsupervised isolation forest algorithm and computing anomaly scores to detect outliers from a 2 dimensional toy dataset. From a scatter plot, I am able to detect/visualize the data points with the highest anomaly scores (example: top 10 or top 15 outliers from the data)as my outliers. Is the number of outliers subjective to user decision (for example:Anomaly score of 0.5>S>1 is an outlier and everything less than 0.5 is not) or is there a way to detect the optimal number of outliers based on the anomaly scores?
unsupervised-learning data-science-model
asked Apr 9 at 18:34
user71207user71207
112
$endgroup$
1
$begingroup$
Hello! Welcome to our community! Would you mind posting your plot, just to illustrate? Maybe we can identify a certain kind of distribution. For example in a Gaussian distribution we consider a sample to be outlier if it differs from the mean by $2sigma$
$endgroup$
– Pedro Henrique Monforte
Apr 9 at 19:01
add a comment |
$begingroup$
I am using an unsupervised isolation forest algorithm and computing anomaly scores to detect outliers from a 2 dimensional toy dataset. From a scatter plot, I am able to detect/visualize the data points with the highest anomaly scores (example: top 10 or top 15 outliers from the data)as my outliers. Is the number of outliers subjective to user decision (for example:Anomaly score of 0.5>S>1 is an outlier and everything less than 0.5 is not) or is there a way to detect the optimal number of outliers based on the anomaly scores?
unsupervised-learning data-science-model
asked Apr 9 at 18:34
user71207user71207
112
$endgroup$
I am using an unsupervised isolation forest algorithm and computing anomaly scores to detect outliers from a 2 dimensional toy dataset. From a scatter plot, I am able to detect/visualize the data points with the highest anomaly scores (example: top 10 or top 15 outliers from the data)as my outliers. Is the number of outliers subjective to user decision (for example:Anomaly score of 0.5>S>1 is an outlier and everything less than 0.5 is not) or is there a way to detect the optimal number of outliers based on the anomaly scores?
unsupervised-learning data-science-model
unsupervised-learning data-science-model
asked Apr 9 at 18:34
user71207user71207
112
asked Apr 9 at 18:34
user71207user71207
112
edited Apr 9 at 19:08
user71207
asked Apr 9 at 18:34
user71207user71207
112
asked Apr 9 at 18:34
user71207user71207
112
asked Apr 9 at 18:34
user71207user71207
112
112
1
$begingroup$
Hello! Welcome to our community! Would you mind posting your plot, just to illustrate? Maybe we can identify a certain kind of distribution. For example in a Gaussian distribution we consider a sample to be outlier if it differs from the mean by $2sigma$
$endgroup$
– Pedro Henrique Monforte
Apr 9 at 19:01
add a comment |
1
$begingroup$
Hello! Welcome to our community! Would you mind posting your plot, just to illustrate? Maybe we can identify a certain kind of distribution. For example in a Gaussian distribution we consider a sample to be outlier if it differs from the mean by $2sigma$
$endgroup$
– Pedro Henrique Monforte
Apr 9 at 19:01
1
1
$begingroup$
Hello! Welcome to our community! Would you mind posting your plot, just to illustrate? Maybe we can identify a certain kind of distribution. For example in a Gaussian distribution we consider a sample to be outlier if it differs from the mean by $2sigma$
$endgroup$
– Pedro Henrique Monforte
Apr 9 at 19:01
$begingroup$
Hello! Welcome to our community! Would you mind posting your plot, just to illustrate? Maybe we can identify a certain kind of distribution. For example in a Gaussian distribution we consider a sample to be outlier if it differs from the mean by $2sigma$
$endgroup$
– Pedro Henrique Monforte
Apr 9 at 19:01
add a comment |
1 Answer
1
active
oldest
votes
$begingroup$
The way Isolation Forests seem to be used in most of the cases involve having some kind of prior "guess" about what proportion of outliers you expect (if you want to be on the safe side, you might be encline to increase it for instance). Based on the proportion you set (it is the contamination parameter in scikit-learn), the observations are labeled according to their anomaly score.
That being said, nothing prevents you from coming up with your own rules after you ran the Isolation Forest. You can plot the anomaly scores and then see for yourself if another threshold could make sense based on the distribution of the scores. This would be a more empirical rule.
answered Apr 10 at 1:43
MaximeKanMaximeKan
1011
$endgroup$
add a comment |
Your Answer
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "557"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f48985%2fusing-an-unsupervised-isolation-forest-how-does-one-identify-the-optimal-number%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
$begingroup$
The way Isolation Forests seem to be used in most of the cases involve having some kind of prior "guess" about what proportion of outliers you expect (if you want to be on the safe side, you might be encline to increase it for instance). Based on the proportion you set (it is the contamination parameter in scikit-learn), the observations are labeled according to their anomaly score.
That being said, nothing prevents you from coming up with your own rules after you ran the Isolation Forest. You can plot the anomaly scores and then see for yourself if another threshold could make sense based on the distribution of the scores. This would be a more empirical rule.
answered Apr 10 at 1:43
MaximeKanMaximeKan
1011
$endgroup$
add a comment |
$begingroup$
The way Isolation Forests seem to be used in most of the cases involve having some kind of prior "guess" about what proportion of outliers you expect (if you want to be on the safe side, you might be encline to increase it for instance). Based on the proportion you set (it is the contamination parameter in scikit-learn), the observations are labeled according to their anomaly score.
That being said, nothing prevents you from coming up with your own rules after you ran the Isolation Forest. You can plot the anomaly scores and then see for yourself if another threshold could make sense based on the distribution of the scores. This would be a more empirical rule.
answered Apr 10 at 1:43
MaximeKanMaximeKan
1011
$endgroup$
add a comment |
$begingroup$
The way Isolation Forests seem to be used in most of the cases involve having some kind of prior "guess" about what proportion of outliers you expect (if you want to be on the safe side, you might be encline to increase it for instance). Based on the proportion you set (it is the contamination parameter in scikit-learn), the observations are labeled according to their anomaly score.
That being said, nothing prevents you from coming up with your own rules after you ran the Isolation Forest. You can plot the anomaly scores and then see for yourself if another threshold could make sense based on the distribution of the scores. This would be a more empirical rule.
answered Apr 10 at 1:43
MaximeKanMaximeKan
1011
$endgroup$
The way Isolation Forests seem to be used in most of the cases involve having some kind of prior "guess" about what proportion of outliers you expect (if you want to be on the safe side, you might be encline to increase it for instance). Based on the proportion you set (it is the contamination parameter in scikit-learn), the observations are labeled according to their anomaly score.
That being said, nothing prevents you from coming up with your own rules after you ran the Isolation Forest. You can plot the anomaly scores and then see for yourself if another threshold could make sense based on the distribution of the scores. This would be a more empirical rule.
answered Apr 10 at 1:43
MaximeKanMaximeKan
1011
answered Apr 10 at 1:43
MaximeKanMaximeKan
1011
answered Apr 10 at 1:43
MaximeKanMaximeKan
1011
answered Apr 10 at 1:43
MaximeKanMaximeKan
1011
1011
add a comment |
add a comment |
Thanks for contributing an answer to Data Science Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
Use MathJax to format equations. MathJax reference.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f48985%2fusing-an-unsupervised-isolation-forest-how-does-one-identify-the-optimal-number%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
1
$begingroup$
Hello! Welcome to our community! Would you mind posting your plot, just to illustrate? Maybe we can identify a certain kind of distribution. For example in a Gaussian distribution we consider a sample to be outlier if it differs from the mean by $2sigma$
$endgroup$
– Pedro Henrique Monforte
Apr 9 at 19:01