what could this mean if your “elbow curve” looks like this?What does 'contextual' mean in 'contextual bandits'?What does “zero-meaned vector” meanWhat did Geoffrey Hinton mean when he said this?What do mean and variance mean for high dimensional data?Is removing poorly predicted data points a valid approach?What does Logits in machine learning mean?What approach other than Tf-Idf could I use for text-clustering using K-Means?What algorithm could be used to fuzzy merge multiple datasets?What is wrong with my Precision-Recall curve?
Unexpected email from Yorkshire Bank
Pulling the rope with one hand is as heavy as with two hands?
Historically, were women trained for obligatory wars? Or did they serve some other military function?
What is the difference between `a[bc]d` (brackets) and `ab,cd` (braces)?
Does the EU Common Fisheries Policy cover British Overseas Territories?
Why is current rating for multicore cable lower than single core with the same cross section?
Please, smoke with good manners
How to creep the reader out with what seems like a normal person?
Minimum value of 4 digit number divided by sum of its digits
How to stop co-workers from teasing me because I know Russian?
Help, my Death Star suffers from Kessler syndrome!
Examples of non trivial equivalence relations , I mean equivalence relations without the expression " same ... as" in their definition?
How does a Swashbuckler rogue "fight with two weapons while safely darting away"?
Find the coordinate of two line segments that are perpendicular
Why does nature favour the Laplacian?
Subtleties of choosing the sequence of tenses in Russian
Sci-fi novel series with instant travel between planets through gates. A river runs through the gates
Feels like I am getting dragged in office politics
Was it really necessary for the Lunar Module to have 2 stages?
Do I have an "anti-research" personality?
Why do computer-science majors learn calculus?
A question regarding using the definite article
Is creating your own "experiment" considered cheating during a physics exam?
Colliding particles and Activation energy
what could this mean if your “elbow curve” looks like this?
What does 'contextual' mean in 'contextual bandits'?What does “zero-meaned vector” meanWhat did Geoffrey Hinton mean when he said this?What do mean and variance mean for high dimensional data?Is removing poorly predicted data points a valid approach?What does Logits in machine learning mean?What approach other than Tf-Idf could I use for text-clustering using K-Means?What algorithm could be used to fuzzy merge multiple datasets?What is wrong with my Precision-Recall curve?
$begingroup$
This is from running kmeans clustering with k on the x-axis (ranging from 2 to 10) and the silhouette distance on the y-axis.
Clearly there's peaks at k=3, k=4 and it seems to decline from there. It doesn't resemble an elbow and thought it should rise as k gets larger (due to over fitting on he training set). Do I just lack data?
I'm computing the silhouette distance using a 80-20 train test split.
machine-learning k-means
asked Apr 8 at 15:18
MrLMrL
83
$endgroup$
add a comment |
$begingroup$
This is from running kmeans clustering with k on the x-axis (ranging from 2 to 10) and the silhouette distance on the y-axis.
Clearly there's peaks at k=3, k=4 and it seems to decline from there. It doesn't resemble an elbow and thought it should rise as k gets larger (due to over fitting on he training set). Do I just lack data?
I'm computing the silhouette distance using a 80-20 train test split.
machine-learning k-means
asked Apr 8 at 15:18
MrLMrL
83
$endgroup$
$begingroup$
So, what’s the size of your data?
$endgroup$
– pythinker
Apr 8 at 20:46
$begingroup$
few thousand rows , TFIDF based clustering ~ 50 000 features
$endgroup$
– MrL
Apr 8 at 21:59
add a comment |
$begingroup$
This is from running kmeans clustering with k on the x-axis (ranging from 2 to 10) and the silhouette distance on the y-axis.
Clearly there's peaks at k=3, k=4 and it seems to decline from there. It doesn't resemble an elbow and thought it should rise as k gets larger (due to over fitting on he training set). Do I just lack data?
I'm computing the silhouette distance using a 80-20 train test split.
machine-learning k-means
asked Apr 8 at 15:18
MrLMrL
83
$endgroup$
This is from running kmeans clustering with k on the x-axis (ranging from 2 to 10) and the silhouette distance on the y-axis.
Clearly there's peaks at k=3, k=4 and it seems to decline from there. It doesn't resemble an elbow and thought it should rise as k gets larger (due to over fitting on he training set). Do I just lack data?
I'm computing the silhouette distance using a 80-20 train test split.
machine-learning k-means
machine-learning k-means
asked Apr 8 at 15:18
MrLMrL
83
asked Apr 8 at 15:18
MrLMrL
83
asked Apr 8 at 15:18
MrLMrL
83
asked Apr 8 at 15:18
MrLMrL
83
asked Apr 8 at 15:18
MrLMrL
83
83
$begingroup$
So, what’s the size of your data?
$endgroup$
– pythinker
Apr 8 at 20:46
$begingroup$
few thousand rows , TFIDF based clustering ~ 50 000 features
$endgroup$
– MrL
Apr 8 at 21:59
add a comment |
$begingroup$
So, what’s the size of your data?
$endgroup$
– pythinker
Apr 8 at 20:46
$begingroup$
few thousand rows , TFIDF based clustering ~ 50 000 features
$endgroup$
– MrL
Apr 8 at 21:59
$begingroup$
So, what’s the size of your data?
$endgroup$
– pythinker
Apr 8 at 20:46
$begingroup$
So, what’s the size of your data?
$endgroup$
– pythinker
Apr 8 at 20:46
$begingroup$
few thousand rows , TFIDF based clustering ~ 50 000 features
$endgroup$
– MrL
Apr 8 at 21:59
$begingroup$
few thousand rows , TFIDF based clustering ~ 50 000 features
$endgroup$
– MrL
Apr 8 at 21:59
add a comment |
1 Answer
1
active
oldest
votes
$begingroup$
First of all, you do have two elbows: one at $k=4$ and a large one at $k=8$. The second isn't very apparent because you haven't drawn out the plot for larger values of $k$. If you do you might get a figure like this:
Secondly, you aren't meant to look for an elbow when computing the silhouette score! The silhouette score accounts for both inter- and intra-cluster distance, as such it can be used for selecting $k$ on its own (i.e. select the $k$ that produces the best silhouette score).
Note: I'm not familiar with the "silhouette distance", I assume it is somewhat related to the silhouette score (maybe its inverse).
The "elbow" criterion should be used when dealing with metrics that tend to improve as $k$ increases (e.g. inertia).
answered Apr 8 at 22:10
Djib2011Djib2011
2,78731225
$endgroup$
add a comment |
Your Answer
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "557"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f48883%2fwhat-could-this-mean-if-your-elbow-curve-looks-like-this%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
$begingroup$
First of all, you do have two elbows: one at $k=4$ and a large one at $k=8$. The second isn't very apparent because you haven't drawn out the plot for larger values of $k$. If you do you might get a figure like this:
Secondly, you aren't meant to look for an elbow when computing the silhouette score! The silhouette score accounts for both inter- and intra-cluster distance, as such it can be used for selecting $k$ on its own (i.e. select the $k$ that produces the best silhouette score).
Note: I'm not familiar with the "silhouette distance", I assume it is somewhat related to the silhouette score (maybe its inverse).
The "elbow" criterion should be used when dealing with metrics that tend to improve as $k$ increases (e.g. inertia).
answered Apr 8 at 22:10
Djib2011Djib2011
2,78731225
$endgroup$
add a comment |
$begingroup$
First of all, you do have two elbows: one at $k=4$ and a large one at $k=8$. The second isn't very apparent because you haven't drawn out the plot for larger values of $k$. If you do you might get a figure like this:
Secondly, you aren't meant to look for an elbow when computing the silhouette score! The silhouette score accounts for both inter- and intra-cluster distance, as such it can be used for selecting $k$ on its own (i.e. select the $k$ that produces the best silhouette score).
Note: I'm not familiar with the "silhouette distance", I assume it is somewhat related to the silhouette score (maybe its inverse).
The "elbow" criterion should be used when dealing with metrics that tend to improve as $k$ increases (e.g. inertia).
answered Apr 8 at 22:10
Djib2011Djib2011
2,78731225
$endgroup$
add a comment |
$begingroup$
First of all, you do have two elbows: one at $k=4$ and a large one at $k=8$. The second isn't very apparent because you haven't drawn out the plot for larger values of $k$. If you do you might get a figure like this:
Secondly, you aren't meant to look for an elbow when computing the silhouette score! The silhouette score accounts for both inter- and intra-cluster distance, as such it can be used for selecting $k$ on its own (i.e. select the $k$ that produces the best silhouette score).
Note: I'm not familiar with the "silhouette distance", I assume it is somewhat related to the silhouette score (maybe its inverse).
The "elbow" criterion should be used when dealing with metrics that tend to improve as $k$ increases (e.g. inertia).
answered Apr 8 at 22:10
Djib2011Djib2011
2,78731225
$endgroup$
First of all, you do have two elbows: one at $k=4$ and a large one at $k=8$. The second isn't very apparent because you haven't drawn out the plot for larger values of $k$. If you do you might get a figure like this:
Secondly, you aren't meant to look for an elbow when computing the silhouette score! The silhouette score accounts for both inter- and intra-cluster distance, as such it can be used for selecting $k$ on its own (i.e. select the $k$ that produces the best silhouette score).
Note: I'm not familiar with the "silhouette distance", I assume it is somewhat related to the silhouette score (maybe its inverse).
The "elbow" criterion should be used when dealing with metrics that tend to improve as $k$ increases (e.g. inertia).
answered Apr 8 at 22:10
Djib2011Djib2011
2,78731225
answered Apr 8 at 22:10
Djib2011Djib2011
2,78731225
answered Apr 8 at 22:10
Djib2011Djib2011
2,78731225
answered Apr 8 at 22:10
Djib2011Djib2011
2,78731225
2,78731225
add a comment |
add a comment |
Thanks for contributing an answer to Data Science Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
Use MathJax to format equations. MathJax reference.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f48883%2fwhat-could-this-mean-if-your-elbow-curve-looks-like-this%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
$begingroup$
So, what’s the size of your data?
$endgroup$
– pythinker
Apr 8 at 20:46
$begingroup$
few thousand rows , TFIDF based clustering ~ 50 000 features
$endgroup$
– MrL
Apr 8 at 21:59