ML regression poor performanceK Means giving poor resultsContrasting logistic regression vs decision tree performance in specific exampleClassification with frequency feature vector produces poor resultsImprove performance of SVMGaussian process regression: sudden increase of the prediction's variancePoor performance of SVM after training for rare eventsRegression model performance with noisy dependent variablebad regression performance on imbalanced dataset
How to balance a monster modification (zombie)?
Do native speakers use "ultima" and "proxima" frequently in spoken English?
How to find the largest number(s) in a list of elements, possibly non-unique?
Unfrosted light bulb
Started in 1987 vs. Starting in 1987
is this saw blade faulty?
What kind of footwear is suitable for walking in micro gravity environment?
What are the consequences of changing the number of hours in a day?
Do I need an EFI partition for each 18.04 ubuntu I have on my HD?
If I cast the Enlarge/Reduce spell on an arrow, what weapon could it count as?
Is this Pascal's Matrix?
What happens when the centripetal force is equal and opposite to the centrifugal force?
Print last inputted byte
"Marked down as someone wanting to sell shares." What does that mean?
When is composition of meromorphic functions meromorphic
How do you justify more code being written by following clean code practices?
Why are there no stars visible in cislunar space?
Why is "la Gestapo" feminine?
The multiplication of list of matrices
Which partition to make active?
label a part of commutative diagram
Exit shell with shortcut (not typing exit) that closes session properly
God... independent
Should a narrator ever describe things based on a characters view instead of fact?
ML regression poor performance
K Means giving poor resultsContrasting logistic regression vs decision tree performance in specific exampleClassification with frequency feature vector produces poor resultsImprove performance of SVMGaussian process regression: sudden increase of the prediction's variancePoor performance of SVM after training for rare eventsRegression model performance with noisy dependent variablebad regression performance on imbalanced dataset
$begingroup$
I am experimenting with 3 years time series electrical demand data (kW) for a building and attempting to create regression supervised ML models from sci kit learn regressor algorithms but I have very poor performance (very high mean squared error). I have a GitHub Gist of the entire IPython notebook here.
There isn't a lot of wisdom here (and I don't have anyone to consult with) for what I am doing other than I know there is well developed analytic software (demand forecasting) that the power consulting industry uses and I am just attempting to recreate from scratch on own experimentation methods in Python.
The data that I am processing look like this below all recorded in 15 minute intervals.
Date_Time kW
0 2011-03-01 00:15:00 171.36
1 2011-03-01 00:30:00 181.44
2 2011-03-01 00:45:00 175.68
3 2011-03-01 01:00:00 180.00
The distribution of the kW data looks like this pic below which doesn't appear to have a bell shaped curve: (Could this be a poor performance reason?)
EDIT rolling average plot
Also in my experimentation I am adding in additional Python Pandas dataframes to represent the integer value of the time stamp 'day of the week', hour, minute, and month; where logically I am know electrical demand fluctuates greatly depending on these variables. These are some scatter plots below of the data compared to kW. (which maybe screwing everything up) For example the first scatter below is the hour of the day which is typical for buildings that the electrical demand increases during a typical work day. The outliers are most likely extreme weather conditions causing high demand where I do not have any weather data incorporated here...
In python if I do a df.describe:
Ultimately I am hoping someone can give me some tips on why the model is horrible but maybe its just due to not enough data and/or strategy... Another person I have been questioning uses a clustering unsupervised learning approach but that doesn't make any sense to me...
Machine learning mastery also has a mini course and a large book I could purchase on time series forecasting. Is this more of a statistics approach? Does it require more 'normal' bell shaped distribution of the data?
Any tips to try or avenues to march down is greatly appreciated :)
EDIT
GitHub gist was updated for a rolling average of the data as well as distribution column plot of kW data
machine-learning python scikit-learn regression supervised-learning
asked Feb 13 at 21:30
HenryHubHenryHub
1567
$endgroup$
bumped to the homepage by Community♦ 2 days ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
add a comment |
$begingroup$
I am experimenting with 3 years time series electrical demand data (kW) for a building and attempting to create regression supervised ML models from sci kit learn regressor algorithms but I have very poor performance (very high mean squared error). I have a GitHub Gist of the entire IPython notebook here.
There isn't a lot of wisdom here (and I don't have anyone to consult with) for what I am doing other than I know there is well developed analytic software (demand forecasting) that the power consulting industry uses and I am just attempting to recreate from scratch on own experimentation methods in Python.
The data that I am processing look like this below all recorded in 15 minute intervals.
Date_Time kW
0 2011-03-01 00:15:00 171.36
1 2011-03-01 00:30:00 181.44
2 2011-03-01 00:45:00 175.68
3 2011-03-01 01:00:00 180.00
The distribution of the kW data looks like this pic below which doesn't appear to have a bell shaped curve: (Could this be a poor performance reason?)
EDIT rolling average plot
Also in my experimentation I am adding in additional Python Pandas dataframes to represent the integer value of the time stamp 'day of the week', hour, minute, and month; where logically I am know electrical demand fluctuates greatly depending on these variables. These are some scatter plots below of the data compared to kW. (which maybe screwing everything up) For example the first scatter below is the hour of the day which is typical for buildings that the electrical demand increases during a typical work day. The outliers are most likely extreme weather conditions causing high demand where I do not have any weather data incorporated here...
In python if I do a df.describe:
Ultimately I am hoping someone can give me some tips on why the model is horrible but maybe its just due to not enough data and/or strategy... Another person I have been questioning uses a clustering unsupervised learning approach but that doesn't make any sense to me...
Machine learning mastery also has a mini course and a large book I could purchase on time series forecasting. Is this more of a statistics approach? Does it require more 'normal' bell shaped distribution of the data?
Any tips to try or avenues to march down is greatly appreciated :)
EDIT
GitHub gist was updated for a rolling average of the data as well as distribution column plot of kW data
machine-learning python scikit-learn regression supervised-learning
asked Feb 13 at 21:30
HenryHubHenryHub
1567
$endgroup$
bumped to the homepage by Community♦ 2 days ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
$begingroup$
Before getting into details, I assume you have fluctuations, on daily basis, then training on the actual data points is quite difficult as it is very noisy! Have you tried to do a rolling mean of "kW" or any other averaging method to reduce those noise a bit? I do not know instead of 15-min data point, maybe average on every 2-hr, 4-hr or so...you gotta try various windows, and see how it improves.
$endgroup$
– Majid Mortazavi
Feb 13 at 21:54
$begingroup$
I’ll try resembling the data to average it per hour.. thanks! Is noise fluctuations in the data that don’t affect the “big picture”?
$endgroup$
– HenryHub
Feb 13 at 22:53
$begingroup$
Yes it wont. Your problem is a a time-series in nature. Basically you have soem of sort of seasonality, trend etc, you have to make your series non-stationary, rolling averaging is one way or up-sampling or.., just google you will find lots of materials. It is only when you have a stable changes in your target, simple models like those you used would give a reasonable result. Good luck.
$endgroup$
– Majid Mortazavi
Feb 14 at 6:48
$begingroup$
I may need to take that "Yes it wont." back, it depends!! Surely some information will be lost, but it helps to generalize.
$endgroup$
– Majid Mortazavi
Feb 14 at 9:09
$begingroup$
@MajidMortazavi thanks for the tips, I updated everything including the Gist for a rolling average... And it didn't improve ML mean squared error much... The distribution of the data plot is a bit less "smooth" looking... The curve almost looks (I think) exponential. Does that have an affect on ML algorithms??
$endgroup$
– HenryHub
Feb 14 at 15:10
add a comment |
$begingroup$
I am experimenting with 3 years time series electrical demand data (kW) for a building and attempting to create regression supervised ML models from sci kit learn regressor algorithms but I have very poor performance (very high mean squared error). I have a GitHub Gist of the entire IPython notebook here.
There isn't a lot of wisdom here (and I don't have anyone to consult with) for what I am doing other than I know there is well developed analytic software (demand forecasting) that the power consulting industry uses and I am just attempting to recreate from scratch on own experimentation methods in Python.
The data that I am processing look like this below all recorded in 15 minute intervals.
Date_Time kW
0 2011-03-01 00:15:00 171.36
1 2011-03-01 00:30:00 181.44
2 2011-03-01 00:45:00 175.68
3 2011-03-01 01:00:00 180.00
The distribution of the kW data looks like this pic below which doesn't appear to have a bell shaped curve: (Could this be a poor performance reason?)
EDIT rolling average plot
Also in my experimentation I am adding in additional Python Pandas dataframes to represent the integer value of the time stamp 'day of the week', hour, minute, and month; where logically I am know electrical demand fluctuates greatly depending on these variables. These are some scatter plots below of the data compared to kW. (which maybe screwing everything up) For example the first scatter below is the hour of the day which is typical for buildings that the electrical demand increases during a typical work day. The outliers are most likely extreme weather conditions causing high demand where I do not have any weather data incorporated here...
In python if I do a df.describe:
Ultimately I am hoping someone can give me some tips on why the model is horrible but maybe its just due to not enough data and/or strategy... Another person I have been questioning uses a clustering unsupervised learning approach but that doesn't make any sense to me...
Machine learning mastery also has a mini course and a large book I could purchase on time series forecasting. Is this more of a statistics approach? Does it require more 'normal' bell shaped distribution of the data?
Any tips to try or avenues to march down is greatly appreciated :)
EDIT
GitHub gist was updated for a rolling average of the data as well as distribution column plot of kW data
machine-learning python scikit-learn regression supervised-learning
asked Feb 13 at 21:30
HenryHubHenryHub
1567
$endgroup$
I am experimenting with 3 years time series electrical demand data (kW) for a building and attempting to create regression supervised ML models from sci kit learn regressor algorithms but I have very poor performance (very high mean squared error). I have a GitHub Gist of the entire IPython notebook here.
There isn't a lot of wisdom here (and I don't have anyone to consult with) for what I am doing other than I know there is well developed analytic software (demand forecasting) that the power consulting industry uses and I am just attempting to recreate from scratch on own experimentation methods in Python.
The data that I am processing look like this below all recorded in 15 minute intervals.
Date_Time kW
0 2011-03-01 00:15:00 171.36
1 2011-03-01 00:30:00 181.44
2 2011-03-01 00:45:00 175.68
3 2011-03-01 01:00:00 180.00
The distribution of the kW data looks like this pic below which doesn't appear to have a bell shaped curve: (Could this be a poor performance reason?)
EDIT rolling average plot
Also in my experimentation I am adding in additional Python Pandas dataframes to represent the integer value of the time stamp 'day of the week', hour, minute, and month; where logically I am know electrical demand fluctuates greatly depending on these variables. These are some scatter plots below of the data compared to kW. (which maybe screwing everything up) For example the first scatter below is the hour of the day which is typical for buildings that the electrical demand increases during a typical work day. The outliers are most likely extreme weather conditions causing high demand where I do not have any weather data incorporated here...
In python if I do a df.describe:
Ultimately I am hoping someone can give me some tips on why the model is horrible but maybe its just due to not enough data and/or strategy... Another person I have been questioning uses a clustering unsupervised learning approach but that doesn't make any sense to me...
Machine learning mastery also has a mini course and a large book I could purchase on time series forecasting. Is this more of a statistics approach? Does it require more 'normal' bell shaped distribution of the data?
Any tips to try or avenues to march down is greatly appreciated :)
EDIT
GitHub gist was updated for a rolling average of the data as well as distribution column plot of kW data
machine-learning python scikit-learn regression supervised-learning
machine-learning python scikit-learn regression supervised-learning
asked Feb 13 at 21:30
HenryHubHenryHub
1567
asked Feb 13 at 21:30
HenryHubHenryHub
1567
edited Feb 14 at 15:03
HenryHub
asked Feb 13 at 21:30
HenryHubHenryHub
1567
asked Feb 13 at 21:30
HenryHubHenryHub
1567
asked Feb 13 at 21:30
HenryHubHenryHub
1567
1567
bumped to the homepage by Community♦ 2 days ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
bumped to the homepage by Community♦ 2 days ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
$begingroup$
Before getting into details, I assume you have fluctuations, on daily basis, then training on the actual data points is quite difficult as it is very noisy! Have you tried to do a rolling mean of "kW" or any other averaging method to reduce those noise a bit? I do not know instead of 15-min data point, maybe average on every 2-hr, 4-hr or so...you gotta try various windows, and see how it improves.
$endgroup$
– Majid Mortazavi
Feb 13 at 21:54
$begingroup$
I’ll try resembling the data to average it per hour.. thanks! Is noise fluctuations in the data that don’t affect the “big picture”?
$endgroup$
– HenryHub
Feb 13 at 22:53
$begingroup$
Yes it wont. Your problem is a a time-series in nature. Basically you have soem of sort of seasonality, trend etc, you have to make your series non-stationary, rolling averaging is one way or up-sampling or.., just google you will find lots of materials. It is only when you have a stable changes in your target, simple models like those you used would give a reasonable result. Good luck.
$endgroup$
– Majid Mortazavi
Feb 14 at 6:48
$begingroup$
I may need to take that "Yes it wont." back, it depends!! Surely some information will be lost, but it helps to generalize.
$endgroup$
– Majid Mortazavi
Feb 14 at 9:09
$begingroup$
@MajidMortazavi thanks for the tips, I updated everything including the Gist for a rolling average... And it didn't improve ML mean squared error much... The distribution of the data plot is a bit less "smooth" looking... The curve almost looks (I think) exponential. Does that have an affect on ML algorithms??
$endgroup$
– HenryHub
Feb 14 at 15:10
add a comment |
$begingroup$
Before getting into details, I assume you have fluctuations, on daily basis, then training on the actual data points is quite difficult as it is very noisy! Have you tried to do a rolling mean of "kW" or any other averaging method to reduce those noise a bit? I do not know instead of 15-min data point, maybe average on every 2-hr, 4-hr or so...you gotta try various windows, and see how it improves.
$endgroup$
– Majid Mortazavi
Feb 13 at 21:54
$begingroup$
I’ll try resembling the data to average it per hour.. thanks! Is noise fluctuations in the data that don’t affect the “big picture”?
$endgroup$
– HenryHub
Feb 13 at 22:53
$begingroup$
Yes it wont. Your problem is a a time-series in nature. Basically you have soem of sort of seasonality, trend etc, you have to make your series non-stationary, rolling averaging is one way or up-sampling or.., just google you will find lots of materials. It is only when you have a stable changes in your target, simple models like those you used would give a reasonable result. Good luck.
$endgroup$
– Majid Mortazavi
Feb 14 at 6:48
$begingroup$
I may need to take that "Yes it wont." back, it depends!! Surely some information will be lost, but it helps to generalize.
$endgroup$
– Majid Mortazavi
Feb 14 at 9:09
$begingroup$
@MajidMortazavi thanks for the tips, I updated everything including the Gist for a rolling average... And it didn't improve ML mean squared error much... The distribution of the data plot is a bit less "smooth" looking... The curve almost looks (I think) exponential. Does that have an affect on ML algorithms??
$endgroup$
– HenryHub
Feb 14 at 15:10
$begingroup$
Before getting into details, I assume you have fluctuations, on daily basis, then training on the actual data points is quite difficult as it is very noisy! Have you tried to do a rolling mean of "kW" or any other averaging method to reduce those noise a bit? I do not know instead of 15-min data point, maybe average on every 2-hr, 4-hr or so...you gotta try various windows, and see how it improves.
$endgroup$
– Majid Mortazavi
Feb 13 at 21:54
$begingroup$
Before getting into details, I assume you have fluctuations, on daily basis, then training on the actual data points is quite difficult as it is very noisy! Have you tried to do a rolling mean of "kW" or any other averaging method to reduce those noise a bit? I do not know instead of 15-min data point, maybe average on every 2-hr, 4-hr or so...you gotta try various windows, and see how it improves.
$endgroup$
– Majid Mortazavi
Feb 13 at 21:54
$begingroup$
I’ll try resembling the data to average it per hour.. thanks! Is noise fluctuations in the data that don’t affect the “big picture”?
$endgroup$
– HenryHub
Feb 13 at 22:53
$begingroup$
I’ll try resembling the data to average it per hour.. thanks! Is noise fluctuations in the data that don’t affect the “big picture”?
$endgroup$
– HenryHub
Feb 13 at 22:53
$begingroup$
Yes it wont. Your problem is a a time-series in nature. Basically you have soem of sort of seasonality, trend etc, you have to make your series non-stationary, rolling averaging is one way or up-sampling or.., just google you will find lots of materials. It is only when you have a stable changes in your target, simple models like those you used would give a reasonable result. Good luck.
$endgroup$
– Majid Mortazavi
Feb 14 at 6:48
$begingroup$
Yes it wont. Your problem is a a time-series in nature. Basically you have soem of sort of seasonality, trend etc, you have to make your series non-stationary, rolling averaging is one way or up-sampling or.., just google you will find lots of materials. It is only when you have a stable changes in your target, simple models like those you used would give a reasonable result. Good luck.
$endgroup$
– Majid Mortazavi
Feb 14 at 6:48
$begingroup$
I may need to take that "Yes it wont." back, it depends!! Surely some information will be lost, but it helps to generalize.
$endgroup$
– Majid Mortazavi
Feb 14 at 9:09
$begingroup$
I may need to take that "Yes it wont." back, it depends!! Surely some information will be lost, but it helps to generalize.
$endgroup$
– Majid Mortazavi
Feb 14 at 9:09
$begingroup$
@MajidMortazavi thanks for the tips, I updated everything including the Gist for a rolling average... And it didn't improve ML mean squared error much... The distribution of the data plot is a bit less "smooth" looking... The curve almost looks (I think) exponential. Does that have an affect on ML algorithms??
$endgroup$
– HenryHub
Feb 14 at 15:10
$begingroup$
@MajidMortazavi thanks for the tips, I updated everything including the Gist for a rolling average... And it didn't improve ML mean squared error much... The distribution of the data plot is a bit less "smooth" looking... The curve almost looks (I think) exponential. Does that have an affect on ML algorithms??
$endgroup$
– HenryHub
Feb 14 at 15:10
add a comment |
1 Answer
1
active
oldest
votes
$begingroup$
It seems to me that your premise for doing this is potentially flawed. It sounds like you're trying to replicate some information that power companies can generate but they are working with more broad datasets than you have presented here. In turn, this can be a reason why your accuracy scores as so poor.
For example, consider that weather has an effect on power usage. So unless you're wrangling in weather data at some point, you will (1) most likely never achieve the most accurate model possible (might just be "adequate") and (2) most likely never approach similar results to what your power company can generate.
So, I would take a step back and consider your current data points; it's highly likely that you just don't have the right factors there in place to create the accurate model you seek.
answered Feb 14 at 19:41
I_Play_With_DataI_Play_With_Data
1,224532
$endgroup$
add a comment |
Your Answer
StackExchange.ifUsing("editor", function ()
return StackExchange.using("mathjaxEditing", function ()
StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix)
StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["$", "$"], ["\\(","\\)"]]);
);
);
, "mathjax-editing");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "557"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f45540%2fml-regression-poor-performance%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
$begingroup$
It seems to me that your premise for doing this is potentially flawed. It sounds like you're trying to replicate some information that power companies can generate but they are working with more broad datasets than you have presented here. In turn, this can be a reason why your accuracy scores as so poor.
For example, consider that weather has an effect on power usage. So unless you're wrangling in weather data at some point, you will (1) most likely never achieve the most accurate model possible (might just be "adequate") and (2) most likely never approach similar results to what your power company can generate.
So, I would take a step back and consider your current data points; it's highly likely that you just don't have the right factors there in place to create the accurate model you seek.
answered Feb 14 at 19:41
I_Play_With_DataI_Play_With_Data
1,224532
$endgroup$
add a comment |
$begingroup$
It seems to me that your premise for doing this is potentially flawed. It sounds like you're trying to replicate some information that power companies can generate but they are working with more broad datasets than you have presented here. In turn, this can be a reason why your accuracy scores as so poor.
For example, consider that weather has an effect on power usage. So unless you're wrangling in weather data at some point, you will (1) most likely never achieve the most accurate model possible (might just be "adequate") and (2) most likely never approach similar results to what your power company can generate.
So, I would take a step back and consider your current data points; it's highly likely that you just don't have the right factors there in place to create the accurate model you seek.
answered Feb 14 at 19:41
I_Play_With_DataI_Play_With_Data
1,224532
$endgroup$
add a comment |
$begingroup$
It seems to me that your premise for doing this is potentially flawed. It sounds like you're trying to replicate some information that power companies can generate but they are working with more broad datasets than you have presented here. In turn, this can be a reason why your accuracy scores as so poor.
For example, consider that weather has an effect on power usage. So unless you're wrangling in weather data at some point, you will (1) most likely never achieve the most accurate model possible (might just be "adequate") and (2) most likely never approach similar results to what your power company can generate.
So, I would take a step back and consider your current data points; it's highly likely that you just don't have the right factors there in place to create the accurate model you seek.
answered Feb 14 at 19:41
I_Play_With_DataI_Play_With_Data
1,224532
$endgroup$
It seems to me that your premise for doing this is potentially flawed. It sounds like you're trying to replicate some information that power companies can generate but they are working with more broad datasets than you have presented here. In turn, this can be a reason why your accuracy scores as so poor.
For example, consider that weather has an effect on power usage. So unless you're wrangling in weather data at some point, you will (1) most likely never achieve the most accurate model possible (might just be "adequate") and (2) most likely never approach similar results to what your power company can generate.
So, I would take a step back and consider your current data points; it's highly likely that you just don't have the right factors there in place to create the accurate model you seek.
answered Feb 14 at 19:41
I_Play_With_DataI_Play_With_Data
1,224532
answered Feb 14 at 19:41
I_Play_With_DataI_Play_With_Data
1,224532
answered Feb 14 at 19:41
I_Play_With_DataI_Play_With_Data
1,224532
answered Feb 14 at 19:41
I_Play_With_DataI_Play_With_Data
1,224532
1,224532
add a comment |
add a comment |
Thanks for contributing an answer to Data Science Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
Use MathJax to format equations. MathJax reference.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f45540%2fml-regression-poor-performance%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
$begingroup$
Before getting into details, I assume you have fluctuations, on daily basis, then training on the actual data points is quite difficult as it is very noisy! Have you tried to do a rolling mean of "kW" or any other averaging method to reduce those noise a bit? I do not know instead of 15-min data point, maybe average on every 2-hr, 4-hr or so...you gotta try various windows, and see how it improves.
$endgroup$
– Majid Mortazavi
Feb 13 at 21:54
$begingroup$
I’ll try resembling the data to average it per hour.. thanks! Is noise fluctuations in the data that don’t affect the “big picture”?
$endgroup$
– HenryHub
Feb 13 at 22:53
$begingroup$
Yes it wont. Your problem is a a time-series in nature. Basically you have soem of sort of seasonality, trend etc, you have to make your series non-stationary, rolling averaging is one way or up-sampling or.., just google you will find lots of materials. It is only when you have a stable changes in your target, simple models like those you used would give a reasonable result. Good luck.
$endgroup$
– Majid Mortazavi
Feb 14 at 6:48
$begingroup$
I may need to take that "Yes it wont." back, it depends!! Surely some information will be lost, but it helps to generalize.
$endgroup$
– Majid Mortazavi
Feb 14 at 9:09
$begingroup$
@MajidMortazavi thanks for the tips, I updated everything including the Gist for a rolling average... And it didn't improve ML mean squared error much... The distribution of the data plot is a bit less "smooth" looking... The curve almost looks (I think) exponential. Does that have an affect on ML algorithms??
$endgroup$
– HenryHub
Feb 14 at 15:10