Data prepration for logistic regression : Value either “not available” or a “year”Logistic Regression implementation does not convergeLogistic regression on biased dataIntuition for Logistic Regression PerformanceUpdate statement for Logistic RegressionSimple logistic regression wrong predictionslogistic regressionThe test of randomness was a logistic regression predicting missingness from all other variablesDealing with NaN (missing) values for Logistic Regression- Best practices?Logistic regression in pythonWhy is logistic regression not sigmoidal?
Open a doc from terminal, but not by its name
Does the UK parliament need to pass secondary legislation to accept the Article 50 extension
Can I still be respawned if I die by falling off the map?
putting logo on same line but after title, latex
Does IPv6 have similar concept of network mask?
When were female captains banned from Starfleet?
Did arcade monitors have same pixel aspect ratio as TV sets?
Does the Linux kernel need a file system to run?
What is the evidence for the "tyranny of the majority problem" in a direct democracy context?
Why should universal income be universal?
Does malloc reserve more space while allocating memory?
Can a stoichiometric mixture of oxygen and methane exist as a liquid at standard pressure and some (low) temperature?
Why Shazam when there is already Superman?
Can disgust be a key component of horror?
Why did the EU agree to delay the Brexit deadline?
15% tax on $7.5k earnings. Is that right?
Can a Canadian Travel to the USA twice, less than 180 days each time?
What should you do when eye contact makes your subordinate uncomfortable?
Why does AES have exactly 10 rounds for a 128-bit key, 12 for 192 bits and 14 for a 256-bit key size?
How to hide some fields of struct in C?
What is the highest possible scrabble score for placing a single tile
Redundant comparison & "if" before assignment
Yosemite Fire Rings - What to Expect?
Quoting Keynes in a lecture
Data prepration for logistic regression : Value either “not available” or a “year”
Logistic Regression implementation does not convergeLogistic regression on biased dataIntuition for Logistic Regression PerformanceUpdate statement for Logistic RegressionSimple logistic regression wrong predictionslogistic regressionThe test of randomness was a logistic regression predicting missingness from all other variablesDealing with NaN (missing) values for Logistic Regression- Best practices?Logistic regression in pythonWhy is logistic regression not sigmoidal?
$begingroup$
I have some data of houses that have been renovated.
In my data there is one column (among others) that captures this information.
It is either "-1" if there has not been yet any renovation, or the information is the year of renovation like "1995" or "2008".
I would like to apply logistic regression.
However, I do not know how to treat this value.
IMHO it looks like a missing value although it is not a missing information.
So, does anybody have an idea how to put these (unordered) values into relation to the ordered years?
On alternative I could think of is binning the information. Like 1990-1995, 1996-2000,...2016-2019.
Any suggestions are highly appreciated.
logistic-regression missing-data
asked Mar 19 at 6:43
toomtoom
1112
New contributor
toom is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
$endgroup$
add a comment |
$begingroup$
I have some data of houses that have been renovated.
In my data there is one column (among others) that captures this information.
It is either "-1" if there has not been yet any renovation, or the information is the year of renovation like "1995" or "2008".
I would like to apply logistic regression.
However, I do not know how to treat this value.
IMHO it looks like a missing value although it is not a missing information.
So, does anybody have an idea how to put these (unordered) values into relation to the ordered years?
On alternative I could think of is binning the information. Like 1990-1995, 1996-2000,...2016-2019.
Any suggestions are highly appreciated.
logistic-regression missing-data
asked Mar 19 at 6:43
toomtoom
1112
New contributor
toom is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
$endgroup$
$begingroup$
If you change value of "year of renovation" to be same "year of construction" for houses that have not been renovated; does it makes sense for this problem ?
$endgroup$
– Shamit Verma
Mar 19 at 7:05
$begingroup$
Yeah, this is a of course a good idea and it also makes sense. In my case this is however, not applicable. Thanks for mentioning
$endgroup$
– toom
Mar 19 at 8:58
add a comment |
$begingroup$
I have some data of houses that have been renovated.
In my data there is one column (among others) that captures this information.
It is either "-1" if there has not been yet any renovation, or the information is the year of renovation like "1995" or "2008".
I would like to apply logistic regression.
However, I do not know how to treat this value.
IMHO it looks like a missing value although it is not a missing information.
So, does anybody have an idea how to put these (unordered) values into relation to the ordered years?
On alternative I could think of is binning the information. Like 1990-1995, 1996-2000,...2016-2019.
Any suggestions are highly appreciated.
logistic-regression missing-data
asked Mar 19 at 6:43
toomtoom
1112
New contributor
toom is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
$endgroup$
I have some data of houses that have been renovated.
In my data there is one column (among others) that captures this information.
It is either "-1" if there has not been yet any renovation, or the information is the year of renovation like "1995" or "2008".
I would like to apply logistic regression.
However, I do not know how to treat this value.
IMHO it looks like a missing value although it is not a missing information.
So, does anybody have an idea how to put these (unordered) values into relation to the ordered years?
On alternative I could think of is binning the information. Like 1990-1995, 1996-2000,...2016-2019.
Any suggestions are highly appreciated.
logistic-regression missing-data
logistic-regression missing-data
asked Mar 19 at 6:43
toomtoom
1112
New contributor
toom is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
asked Mar 19 at 6:43
toomtoom
1112
New contributor
toom is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
asked Mar 19 at 6:43
toomtoom
1112
New contributor
toom is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
asked Mar 19 at 6:43
toomtoom
1112
asked Mar 19 at 6:43
toomtoom
1112
1112
New contributor
toom is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
New contributor
toom is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
toom is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
$begingroup$
If you change value of "year of renovation" to be same "year of construction" for houses that have not been renovated; does it makes sense for this problem ?
$endgroup$
– Shamit Verma
Mar 19 at 7:05
$begingroup$
Yeah, this is a of course a good idea and it also makes sense. In my case this is however, not applicable. Thanks for mentioning
$endgroup$
– toom
Mar 19 at 8:58
add a comment |
$begingroup$
If you change value of "year of renovation" to be same "year of construction" for houses that have not been renovated; does it makes sense for this problem ?
$endgroup$
– Shamit Verma
Mar 19 at 7:05
$begingroup$
Yeah, this is a of course a good idea and it also makes sense. In my case this is however, not applicable. Thanks for mentioning
$endgroup$
– toom
Mar 19 at 8:58
$begingroup$
If you change value of "year of renovation" to be same "year of construction" for houses that have not been renovated; does it makes sense for this problem ?
$endgroup$
– Shamit Verma
Mar 19 at 7:05
$begingroup$
If you change value of "year of renovation" to be same "year of construction" for houses that have not been renovated; does it makes sense for this problem ?
$endgroup$
– Shamit Verma
Mar 19 at 7:05
$begingroup$
Yeah, this is a of course a good idea and it also makes sense. In my case this is however, not applicable. Thanks for mentioning
$endgroup$
– toom
Mar 19 at 8:58
$begingroup$
Yeah, this is a of course a good idea and it also makes sense. In my case this is however, not applicable. Thanks for mentioning
$endgroup$
– toom
Mar 19 at 8:58
add a comment |
1 Answer
1
active
oldest
votes
$begingroup$
First use a binary 0 (no renovation) and 1 (renovation) which works perfect with logistic regression.
Using the exact date is a bad practice. It guides the model in the direction of over-fitting on specific dates. For example, a pattern from 2006 would be specific to that year and would not help the future years. As an alternative, binning on larger spans like 5 years, 10 years (depends on the context) seems as an improvement. For example:
bins = [1990, 2000], [2000, 2010], [2010, 2020]
[1990, 2000] $rightarrow$ (1, 0, 0)
[2000, 2010] $rightarrow$ (0, 1, 0)
[2010, 2020] $rightarrow$ (0, 0, 1)
This approach also has a tendency to over-fit but over a larger time span. Also note that, this way, your model always has an expiration date, since if we pass the last bin in 2021, there is no bin to cover the year. And if we include [2020, 2030] now, there is no data to learn about this bin. And using [2020, forever] is equally useless for future.
I suggest using the age of construction and renovation which are generalizable. A 5 years old house in 2000 could help us infer about a 5 years old house in 2010, 2020, or 2030. For houses with no renovation, age could be set to -1, which works fine with logistic regression (experiment with 0 too). So as a final example:
renovation (has renovation, renovation age)
-1 (0, -1)
2010 in 2019 (1, 9)
Note that repetitive time features are OK. For example, "Spring", "Monday", or "8:00PM", etc.
answered Mar 19 at 7:52
EsmailianEsmailian
1,686114
$endgroup$
$begingroup$
Great advice. Thanks for this help. Makes sense :)
$endgroup$
– toom
Mar 19 at 9:00
add a comment |
Your Answer
StackExchange.ifUsing("editor", function ()
return StackExchange.using("mathjaxEditing", function ()
StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix)
StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["$", "$"], ["\\(","\\)"]]);
);
);
, "mathjax-editing");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "557"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
toom is a new contributor. Be nice, and check out our Code of Conduct.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f47586%2fdata-prepration-for-logistic-regression-value-either-not-available-or-a-yea%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
$begingroup$
First use a binary 0 (no renovation) and 1 (renovation) which works perfect with logistic regression.
Using the exact date is a bad practice. It guides the model in the direction of over-fitting on specific dates. For example, a pattern from 2006 would be specific to that year and would not help the future years. As an alternative, binning on larger spans like 5 years, 10 years (depends on the context) seems as an improvement. For example:
bins = [1990, 2000], [2000, 2010], [2010, 2020]
[1990, 2000] $rightarrow$ (1, 0, 0)
[2000, 2010] $rightarrow$ (0, 1, 0)
[2010, 2020] $rightarrow$ (0, 0, 1)
This approach also has a tendency to over-fit but over a larger time span. Also note that, this way, your model always has an expiration date, since if we pass the last bin in 2021, there is no bin to cover the year. And if we include [2020, 2030] now, there is no data to learn about this bin. And using [2020, forever] is equally useless for future.
I suggest using the age of construction and renovation which are generalizable. A 5 years old house in 2000 could help us infer about a 5 years old house in 2010, 2020, or 2030. For houses with no renovation, age could be set to -1, which works fine with logistic regression (experiment with 0 too). So as a final example:
renovation (has renovation, renovation age)
-1 (0, -1)
2010 in 2019 (1, 9)
Note that repetitive time features are OK. For example, "Spring", "Monday", or "8:00PM", etc.
answered Mar 19 at 7:52
EsmailianEsmailian
1,686114
$endgroup$
$begingroup$
Great advice. Thanks for this help. Makes sense :)
$endgroup$
– toom
Mar 19 at 9:00
add a comment |
$begingroup$
First use a binary 0 (no renovation) and 1 (renovation) which works perfect with logistic regression.
Using the exact date is a bad practice. It guides the model in the direction of over-fitting on specific dates. For example, a pattern from 2006 would be specific to that year and would not help the future years. As an alternative, binning on larger spans like 5 years, 10 years (depends on the context) seems as an improvement. For example:
bins = [1990, 2000], [2000, 2010], [2010, 2020]
[1990, 2000] $rightarrow$ (1, 0, 0)
[2000, 2010] $rightarrow$ (0, 1, 0)
[2010, 2020] $rightarrow$ (0, 0, 1)
This approach also has a tendency to over-fit but over a larger time span. Also note that, this way, your model always has an expiration date, since if we pass the last bin in 2021, there is no bin to cover the year. And if we include [2020, 2030] now, there is no data to learn about this bin. And using [2020, forever] is equally useless for future.
I suggest using the age of construction and renovation which are generalizable. A 5 years old house in 2000 could help us infer about a 5 years old house in 2010, 2020, or 2030. For houses with no renovation, age could be set to -1, which works fine with logistic regression (experiment with 0 too). So as a final example:
renovation (has renovation, renovation age)
-1 (0, -1)
2010 in 2019 (1, 9)
Note that repetitive time features are OK. For example, "Spring", "Monday", or "8:00PM", etc.
answered Mar 19 at 7:52
EsmailianEsmailian
1,686114
$endgroup$
$begingroup$
Great advice. Thanks for this help. Makes sense :)
$endgroup$
– toom
Mar 19 at 9:00
add a comment |
$begingroup$
First use a binary 0 (no renovation) and 1 (renovation) which works perfect with logistic regression.
Using the exact date is a bad practice. It guides the model in the direction of over-fitting on specific dates. For example, a pattern from 2006 would be specific to that year and would not help the future years. As an alternative, binning on larger spans like 5 years, 10 years (depends on the context) seems as an improvement. For example:
bins = [1990, 2000], [2000, 2010], [2010, 2020]
[1990, 2000] $rightarrow$ (1, 0, 0)
[2000, 2010] $rightarrow$ (0, 1, 0)
[2010, 2020] $rightarrow$ (0, 0, 1)
This approach also has a tendency to over-fit but over a larger time span. Also note that, this way, your model always has an expiration date, since if we pass the last bin in 2021, there is no bin to cover the year. And if we include [2020, 2030] now, there is no data to learn about this bin. And using [2020, forever] is equally useless for future.
I suggest using the age of construction and renovation which are generalizable. A 5 years old house in 2000 could help us infer about a 5 years old house in 2010, 2020, or 2030. For houses with no renovation, age could be set to -1, which works fine with logistic regression (experiment with 0 too). So as a final example:
renovation (has renovation, renovation age)
-1 (0, -1)
2010 in 2019 (1, 9)
Note that repetitive time features are OK. For example, "Spring", "Monday", or "8:00PM", etc.
answered Mar 19 at 7:52
EsmailianEsmailian
1,686114
$endgroup$
First use a binary 0 (no renovation) and 1 (renovation) which works perfect with logistic regression.
Using the exact date is a bad practice. It guides the model in the direction of over-fitting on specific dates. For example, a pattern from 2006 would be specific to that year and would not help the future years. As an alternative, binning on larger spans like 5 years, 10 years (depends on the context) seems as an improvement. For example:
bins = [1990, 2000], [2000, 2010], [2010, 2020]
[1990, 2000] $rightarrow$ (1, 0, 0)
[2000, 2010] $rightarrow$ (0, 1, 0)
[2010, 2020] $rightarrow$ (0, 0, 1)
This approach also has a tendency to over-fit but over a larger time span. Also note that, this way, your model always has an expiration date, since if we pass the last bin in 2021, there is no bin to cover the year. And if we include [2020, 2030] now, there is no data to learn about this bin. And using [2020, forever] is equally useless for future.
I suggest using the age of construction and renovation which are generalizable. A 5 years old house in 2000 could help us infer about a 5 years old house in 2010, 2020, or 2030. For houses with no renovation, age could be set to -1, which works fine with logistic regression (experiment with 0 too). So as a final example:
renovation (has renovation, renovation age)
-1 (0, -1)
2010 in 2019 (1, 9)
Note that repetitive time features are OK. For example, "Spring", "Monday", or "8:00PM", etc.
answered Mar 19 at 7:52
EsmailianEsmailian
1,686114
edited Mar 19 at 12:32
answered Mar 19 at 7:52
EsmailianEsmailian
1,686114
answered Mar 19 at 7:52
EsmailianEsmailian
1,686114
answered Mar 19 at 7:52
EsmailianEsmailian
1,686114
1,686114
$begingroup$
Great advice. Thanks for this help. Makes sense :)
$endgroup$
– toom
Mar 19 at 9:00
add a comment |
$begingroup$
Great advice. Thanks for this help. Makes sense :)
$endgroup$
– toom
Mar 19 at 9:00
$begingroup$
Great advice. Thanks for this help. Makes sense :)
$endgroup$
– toom
Mar 19 at 9:00
$begingroup$
Great advice. Thanks for this help. Makes sense :)
$endgroup$
– toom
Mar 19 at 9:00
add a comment |
toom is a new contributor. Be nice, and check out our Code of Conduct.
toom is a new contributor. Be nice, and check out our Code of Conduct.
toom is a new contributor. Be nice, and check out our Code of Conduct.
toom is a new contributor. Be nice, and check out our Code of Conduct.
Thanks for contributing an answer to Data Science Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
Use MathJax to format equations. MathJax reference.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f47586%2fdata-prepration-for-logistic-regression-value-either-not-available-or-a-yea%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
$begingroup$
If you change value of "year of renovation" to be same "year of construction" for houses that have not been renovated; does it makes sense for this problem ?
$endgroup$
– Shamit Verma
Mar 19 at 7:05
$begingroup$
Yeah, this is a of course a good idea and it also makes sense. In my case this is however, not applicable. Thanks for mentioning
$endgroup$
– toom
Mar 19 at 8:58