Data prepration for logistic regression : Value either “not available” or a “year”Logistic Regression implementation does not convergeLogistic regression on biased dataIntuition for Logistic Regression PerformanceUpdate statement for Logistic RegressionSimple logistic regression wrong predictionslogistic regressionThe test of randomness was a logistic regression predicting missingness from all other variablesDealing with NaN (missing) values for Logistic Regression- Best practices?Logistic regression in pythonWhy is logistic regression not sigmoidal?
Why does a 97 / 92 key piano exist by Bosendorfer?
What can I do if I am asked to learn different programming languages very frequently?
Are hand made posters acceptable in Academia?
Highest stage count that are used one right after the other?
How do I lift the insulation blower into the attic?
Is there any common country to visit for persons holding UK and Schengen visas?
C++ lambda syntax
categorizing a variable turns it from insignificant to significant
A seasonal riddle
What is it called when someone votes for an option that's not their first choice?
1 John in Luther’s Bibel
Friend wants my recommendation but I don't want to give it to him
What is this high flying aircraft over Pennsylvania?
Center page as a whole without centering each element individually
Derivative of an interpolated function
Can you describe someone as luxurious? As in someone who likes luxurious things?
Would this string work as string?
What is the meaning of "You've never met a graph you didn't like?"
Calculate Pi using Monte Carlo
Are all namekians brothers?
Weird lines in Microsoft Word
Turning a hard to access nut?
Why is indicated airspeed rather than ground speed used during the takeoff roll?
Magnifying glass in hyperbolic space
Data prepration for logistic regression : Value either “not available” or a “year”
Logistic Regression implementation does not convergeLogistic regression on biased dataIntuition for Logistic Regression PerformanceUpdate statement for Logistic RegressionSimple logistic regression wrong predictionslogistic regressionThe test of randomness was a logistic regression predicting missingness from all other variablesDealing with NaN (missing) values for Logistic Regression- Best practices?Logistic regression in pythonWhy is logistic regression not sigmoidal?
$begingroup$
I have some data of houses that have been renovated.
In my data there is one column (among others) that captures this information.
It is either "-1" if there has not been yet any renovation, or the information is the year of renovation like "1995" or "2008".
I would like to apply logistic regression.
However, I do not know how to treat this value.
IMHO it looks like a missing value although it is not a missing information.
So, does anybody have an idea how to put these (unordered) values into relation to the ordered years?
On alternative I could think of is binning the information. Like 1990-1995, 1996-2000,...2016-2019.
Any suggestions are highly appreciated.
logistic-regression missing-data
asked 22 hours ago
toomtoom
1112
New contributor
toom is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
$endgroup$
add a comment |
$begingroup$
I have some data of houses that have been renovated.
In my data there is one column (among others) that captures this information.
It is either "-1" if there has not been yet any renovation, or the information is the year of renovation like "1995" or "2008".
I would like to apply logistic regression.
However, I do not know how to treat this value.
IMHO it looks like a missing value although it is not a missing information.
So, does anybody have an idea how to put these (unordered) values into relation to the ordered years?
On alternative I could think of is binning the information. Like 1990-1995, 1996-2000,...2016-2019.
Any suggestions are highly appreciated.
logistic-regression missing-data
asked 22 hours ago
toomtoom
1112
New contributor
toom is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
$endgroup$
$begingroup$
If you change value of "year of renovation" to be same "year of construction" for houses that have not been renovated; does it makes sense for this problem ?
$endgroup$
– Shamit Verma
21 hours ago
$begingroup$
Yeah, this is a of course a good idea and it also makes sense. In my case this is however, not applicable. Thanks for mentioning
$endgroup$
– toom
19 hours ago
add a comment |
$begingroup$
I have some data of houses that have been renovated.
In my data there is one column (among others) that captures this information.
It is either "-1" if there has not been yet any renovation, or the information is the year of renovation like "1995" or "2008".
I would like to apply logistic regression.
However, I do not know how to treat this value.
IMHO it looks like a missing value although it is not a missing information.
So, does anybody have an idea how to put these (unordered) values into relation to the ordered years?
On alternative I could think of is binning the information. Like 1990-1995, 1996-2000,...2016-2019.
Any suggestions are highly appreciated.
logistic-regression missing-data
asked 22 hours ago
toomtoom
1112
New contributor
toom is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
$endgroup$
I have some data of houses that have been renovated.
In my data there is one column (among others) that captures this information.
It is either "-1" if there has not been yet any renovation, or the information is the year of renovation like "1995" or "2008".
I would like to apply logistic regression.
However, I do not know how to treat this value.
IMHO it looks like a missing value although it is not a missing information.
So, does anybody have an idea how to put these (unordered) values into relation to the ordered years?
On alternative I could think of is binning the information. Like 1990-1995, 1996-2000,...2016-2019.
Any suggestions are highly appreciated.
logistic-regression missing-data
logistic-regression missing-data
asked 22 hours ago
toomtoom
1112
New contributor
toom is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
asked 22 hours ago
toomtoom
1112
New contributor
toom is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
asked 22 hours ago
toomtoom
1112
New contributor
toom is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
asked 22 hours ago
toomtoom
1112
asked 22 hours ago
toomtoom
1112
1112
New contributor
toom is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
New contributor
toom is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
toom is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
$begingroup$
If you change value of "year of renovation" to be same "year of construction" for houses that have not been renovated; does it makes sense for this problem ?
$endgroup$
– Shamit Verma
21 hours ago
$begingroup$
Yeah, this is a of course a good idea and it also makes sense. In my case this is however, not applicable. Thanks for mentioning
$endgroup$
– toom
19 hours ago
add a comment |
$begingroup$
If you change value of "year of renovation" to be same "year of construction" for houses that have not been renovated; does it makes sense for this problem ?
$endgroup$
– Shamit Verma
21 hours ago
$begingroup$
Yeah, this is a of course a good idea and it also makes sense. In my case this is however, not applicable. Thanks for mentioning
$endgroup$
– toom
19 hours ago
$begingroup$
If you change value of "year of renovation" to be same "year of construction" for houses that have not been renovated; does it makes sense for this problem ?
$endgroup$
– Shamit Verma
21 hours ago
$begingroup$
If you change value of "year of renovation" to be same "year of construction" for houses that have not been renovated; does it makes sense for this problem ?
$endgroup$
– Shamit Verma
21 hours ago
$begingroup$
Yeah, this is a of course a good idea and it also makes sense. In my case this is however, not applicable. Thanks for mentioning
$endgroup$
– toom
19 hours ago
$begingroup$
Yeah, this is a of course a good idea and it also makes sense. In my case this is however, not applicable. Thanks for mentioning
$endgroup$
– toom
19 hours ago
add a comment |
1 Answer
1
active
oldest
votes
$begingroup$
First use a binary 0 (no renovation) and 1 (renovation) which works perfect with logistic regression.
Using the exact date is a bad practice. It guides the model in the direction of over-fitting on specific dates. For example, a pattern from 2006 would be specific to that year and would not help the future years. As an alternative, binning on larger spans like 5 years, 10 years (depends on the context) seems as an improvement. For example:
bins = [1990, 2000], [2000, 2010], [2010, 2020]
[1990, 2000] $rightarrow$ (1, 0, 0)
[2000, 2010] $rightarrow$ (0, 1, 0)
[2010, 2020] $rightarrow$ (0, 0, 1)
This approach also has a tendency to over-fit but over a larger time span. Also note that, this way, your model always has an expiration date, since if we pass the last bin in 2021, there is no bin to cover the year. And if we include [2020, 2030] now, there is no data to learn about this bin. And using [2020, forever] is equally useless for future.
I suggest using the age of construction and renovation which are generalizable. A 5 years old house in 2000 could help us infer about a 5 years old house in 2010, 2020, or 2030. For houses with no renovation, age could be set to -1, which works fine with logistic regression (experiment with 0 too). So as a final example:
renovation (has renovation, renovation age)
-1 (0, -1)
2010 in 2019 (1, 9)
Note that repetitive time features are OK. For example, "Spring", "Monday", or "8:00PM", etc.
answered 21 hours ago
EsmailianEsmailian
1,536113
$endgroup$
$begingroup$
Great advice. Thanks for this help. Makes sense :)
$endgroup$
– toom
19 hours ago
add a comment |
Your Answer
StackExchange.ifUsing("editor", function ()
return StackExchange.using("mathjaxEditing", function ()
StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix)
StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["$", "$"], ["\\(","\\)"]]);
);
);
, "mathjax-editing");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "557"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
toom is a new contributor. Be nice, and check out our Code of Conduct.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f47586%2fdata-prepration-for-logistic-regression-value-either-not-available-or-a-yea%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
$begingroup$
First use a binary 0 (no renovation) and 1 (renovation) which works perfect with logistic regression.
Using the exact date is a bad practice. It guides the model in the direction of over-fitting on specific dates. For example, a pattern from 2006 would be specific to that year and would not help the future years. As an alternative, binning on larger spans like 5 years, 10 years (depends on the context) seems as an improvement. For example:
bins = [1990, 2000], [2000, 2010], [2010, 2020]
[1990, 2000] $rightarrow$ (1, 0, 0)
[2000, 2010] $rightarrow$ (0, 1, 0)
[2010, 2020] $rightarrow$ (0, 0, 1)
This approach also has a tendency to over-fit but over a larger time span. Also note that, this way, your model always has an expiration date, since if we pass the last bin in 2021, there is no bin to cover the year. And if we include [2020, 2030] now, there is no data to learn about this bin. And using [2020, forever] is equally useless for future.
I suggest using the age of construction and renovation which are generalizable. A 5 years old house in 2000 could help us infer about a 5 years old house in 2010, 2020, or 2030. For houses with no renovation, age could be set to -1, which works fine with logistic regression (experiment with 0 too). So as a final example:
renovation (has renovation, renovation age)
-1 (0, -1)
2010 in 2019 (1, 9)
Note that repetitive time features are OK. For example, "Spring", "Monday", or "8:00PM", etc.
answered 21 hours ago
EsmailianEsmailian
1,536113
$endgroup$
$begingroup$
Great advice. Thanks for this help. Makes sense :)
$endgroup$
– toom
19 hours ago
add a comment |
$begingroup$
First use a binary 0 (no renovation) and 1 (renovation) which works perfect with logistic regression.
Using the exact date is a bad practice. It guides the model in the direction of over-fitting on specific dates. For example, a pattern from 2006 would be specific to that year and would not help the future years. As an alternative, binning on larger spans like 5 years, 10 years (depends on the context) seems as an improvement. For example:
bins = [1990, 2000], [2000, 2010], [2010, 2020]
[1990, 2000] $rightarrow$ (1, 0, 0)
[2000, 2010] $rightarrow$ (0, 1, 0)
[2010, 2020] $rightarrow$ (0, 0, 1)
This approach also has a tendency to over-fit but over a larger time span. Also note that, this way, your model always has an expiration date, since if we pass the last bin in 2021, there is no bin to cover the year. And if we include [2020, 2030] now, there is no data to learn about this bin. And using [2020, forever] is equally useless for future.
I suggest using the age of construction and renovation which are generalizable. A 5 years old house in 2000 could help us infer about a 5 years old house in 2010, 2020, or 2030. For houses with no renovation, age could be set to -1, which works fine with logistic regression (experiment with 0 too). So as a final example:
renovation (has renovation, renovation age)
-1 (0, -1)
2010 in 2019 (1, 9)
Note that repetitive time features are OK. For example, "Spring", "Monday", or "8:00PM", etc.
answered 21 hours ago
EsmailianEsmailian
1,536113
$endgroup$
$begingroup$
Great advice. Thanks for this help. Makes sense :)
$endgroup$
– toom
19 hours ago
add a comment |
$begingroup$
First use a binary 0 (no renovation) and 1 (renovation) which works perfect with logistic regression.
Using the exact date is a bad practice. It guides the model in the direction of over-fitting on specific dates. For example, a pattern from 2006 would be specific to that year and would not help the future years. As an alternative, binning on larger spans like 5 years, 10 years (depends on the context) seems as an improvement. For example:
bins = [1990, 2000], [2000, 2010], [2010, 2020]
[1990, 2000] $rightarrow$ (1, 0, 0)
[2000, 2010] $rightarrow$ (0, 1, 0)
[2010, 2020] $rightarrow$ (0, 0, 1)
This approach also has a tendency to over-fit but over a larger time span. Also note that, this way, your model always has an expiration date, since if we pass the last bin in 2021, there is no bin to cover the year. And if we include [2020, 2030] now, there is no data to learn about this bin. And using [2020, forever] is equally useless for future.
I suggest using the age of construction and renovation which are generalizable. A 5 years old house in 2000 could help us infer about a 5 years old house in 2010, 2020, or 2030. For houses with no renovation, age could be set to -1, which works fine with logistic regression (experiment with 0 too). So as a final example:
renovation (has renovation, renovation age)
-1 (0, -1)
2010 in 2019 (1, 9)
Note that repetitive time features are OK. For example, "Spring", "Monday", or "8:00PM", etc.
answered 21 hours ago
EsmailianEsmailian
1,536113
$endgroup$
First use a binary 0 (no renovation) and 1 (renovation) which works perfect with logistic regression.
Using the exact date is a bad practice. It guides the model in the direction of over-fitting on specific dates. For example, a pattern from 2006 would be specific to that year and would not help the future years. As an alternative, binning on larger spans like 5 years, 10 years (depends on the context) seems as an improvement. For example:
bins = [1990, 2000], [2000, 2010], [2010, 2020]
[1990, 2000] $rightarrow$ (1, 0, 0)
[2000, 2010] $rightarrow$ (0, 1, 0)
[2010, 2020] $rightarrow$ (0, 0, 1)
This approach also has a tendency to over-fit but over a larger time span. Also note that, this way, your model always has an expiration date, since if we pass the last bin in 2021, there is no bin to cover the year. And if we include [2020, 2030] now, there is no data to learn about this bin. And using [2020, forever] is equally useless for future.
I suggest using the age of construction and renovation which are generalizable. A 5 years old house in 2000 could help us infer about a 5 years old house in 2010, 2020, or 2030. For houses with no renovation, age could be set to -1, which works fine with logistic regression (experiment with 0 too). So as a final example:
renovation (has renovation, renovation age)
-1 (0, -1)
2010 in 2019 (1, 9)
Note that repetitive time features are OK. For example, "Spring", "Monday", or "8:00PM", etc.
answered 21 hours ago
EsmailianEsmailian
1,536113
edited 16 hours ago
answered 21 hours ago
EsmailianEsmailian
1,536113
answered 21 hours ago
EsmailianEsmailian
1,536113
answered 21 hours ago
EsmailianEsmailian
1,536113
1,536113
$begingroup$
Great advice. Thanks for this help. Makes sense :)
$endgroup$
– toom
19 hours ago
add a comment |
$begingroup$
Great advice. Thanks for this help. Makes sense :)
$endgroup$
– toom
19 hours ago
$begingroup$
Great advice. Thanks for this help. Makes sense :)
$endgroup$
– toom
19 hours ago
$begingroup$
Great advice. Thanks for this help. Makes sense :)
$endgroup$
– toom
19 hours ago
add a comment |
toom is a new contributor. Be nice, and check out our Code of Conduct.
toom is a new contributor. Be nice, and check out our Code of Conduct.
toom is a new contributor. Be nice, and check out our Code of Conduct.
toom is a new contributor. Be nice, and check out our Code of Conduct.
Thanks for contributing an answer to Data Science Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
Use MathJax to format equations. MathJax reference.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f47586%2fdata-prepration-for-logistic-regression-value-either-not-available-or-a-yea%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
$begingroup$
If you change value of "year of renovation" to be same "year of construction" for houses that have not been renovated; does it makes sense for this problem ?
$endgroup$
– Shamit Verma
21 hours ago
$begingroup$
Yeah, this is a of course a good idea and it also makes sense. In my case this is however, not applicable. Thanks for mentioning
$endgroup$
– toom
19 hours ago