Can I accurately call sklearn.model_selection.train_test_split multiple times when data doesn't fit into memory?2019 Community Moderator ElectionTechnical name for this data wrangling process? Multiple columns into multi-factor single columnPredictive modeling on big data set that can't fit into memoryHow can l get 50 % examples in training set and 50% in test set for each class when splitting data?SVM is not fitted when tried to fit it into a modelHow can I plot data after cluster it into two cluster?Data matching between different columns when multiple values in single row
Are the number of citations and number of published articles the most important criteria for a tenure promotion?
Rock identification in KY
expand `ifthenelse` immediately
A newer friend of my brother's gave him a load of baseball cards that are supposedly extremely valuable. Is this a scam?
How much of data wrangling is a data scientist's job?
Was any UN Security Council vote triple-vetoed?
How to format long polynomial?
Did Shadowfax go to Valinor?
Uncaught TypeError: 'set' on proxy: trap returned falsish for property Name
Why doesn't Newton's third law mean a person bounces back to where they started when they hit the ground?
Accidentally leaked the solution to an assignment, what to do now? (I'm the prof)
Why do I get two different answers for this counting problem?
Is it possible to do 50 km distance without any previous training?
Has there ever been an airliner design involving reducing generator load by installing solar panels?
Client team has low performances and low technical skills: we always fix their work and now they stop collaborate with us. How to solve?
Why is consensus so controversial in Britain?
Why does Kotter return in Welcome Back Kotter?
What would happen to a modern skyscraper if it rains micro blackholes?
Mutually beneficial digestive system symbiotes
What typically incentivizes a professor to change jobs to a lower ranking university?
dbcc cleantable batch size explanation
High voltage LED indicator 40-1000 VDC without additional power supply
Can a monk's single staff be considered dual wielded, as per the Dual Wielder feat?
Decision tree nodes overlapping with Tikz
Can I accurately call sklearn.model_selection.train_test_split multiple times when data doesn't fit into memory?
2019 Community Moderator ElectionTechnical name for this data wrangling process? Multiple columns into multi-factor single columnPredictive modeling on big data set that can't fit into memoryHow can l get 50 % examples in training set and 50% in test set for each class when splitting data?SVM is not fitted when tried to fit it into a modelHow can I plot data after cluster it into two cluster?Data matching between different columns when multiple values in single row
$begingroup$
Consider a very large data set that doesn't fit into memory. Would I be able to get (nearly) the same behavior from multiple calls to train_test_split when calling train_test_split by passing batches of a source data set as opposed to the whole thing at once?
This code is just hypothetical to illustrate my question.
# X, y is the entire dataset.
x_train, y_train, x_test, y_test = train_test_split(X,y,stratify=y, test_size=.2)
# compared to
for x_bat, y_bat in stream_next_batch_from_file():
x_train, y_train, x_test, y_test = train_test_split(x_bat, y_bat, stratify=y_bat, test_size=.2)
# Append the splits to their respective files.
append_data(x_train, y_train, "train_set_filename")
append_data(x_test, y_test, "test_set_filename")
# etc.
scikit-learn dataset data
asked Mar 27 at 21:57
Joey CarsonJoey Carson
1084
$endgroup$
add a comment |
$begingroup$
Consider a very large data set that doesn't fit into memory. Would I be able to get (nearly) the same behavior from multiple calls to train_test_split when calling train_test_split by passing batches of a source data set as opposed to the whole thing at once?
This code is just hypothetical to illustrate my question.
# X, y is the entire dataset.
x_train, y_train, x_test, y_test = train_test_split(X,y,stratify=y, test_size=.2)
# compared to
for x_bat, y_bat in stream_next_batch_from_file():
x_train, y_train, x_test, y_test = train_test_split(x_bat, y_bat, stratify=y_bat, test_size=.2)
# Append the splits to their respective files.
append_data(x_train, y_train, "train_set_filename")
append_data(x_test, y_test, "test_set_filename")
# etc.
scikit-learn dataset data
asked Mar 27 at 21:57
Joey CarsonJoey Carson
1084
$endgroup$
add a comment |
$begingroup$
Consider a very large data set that doesn't fit into memory. Would I be able to get (nearly) the same behavior from multiple calls to train_test_split when calling train_test_split by passing batches of a source data set as opposed to the whole thing at once?
This code is just hypothetical to illustrate my question.
# X, y is the entire dataset.
x_train, y_train, x_test, y_test = train_test_split(X,y,stratify=y, test_size=.2)
# compared to
for x_bat, y_bat in stream_next_batch_from_file():
x_train, y_train, x_test, y_test = train_test_split(x_bat, y_bat, stratify=y_bat, test_size=.2)
# Append the splits to their respective files.
append_data(x_train, y_train, "train_set_filename")
append_data(x_test, y_test, "test_set_filename")
# etc.
scikit-learn dataset data
asked Mar 27 at 21:57
Joey CarsonJoey Carson
1084
$endgroup$
Consider a very large data set that doesn't fit into memory. Would I be able to get (nearly) the same behavior from multiple calls to train_test_split when calling train_test_split by passing batches of a source data set as opposed to the whole thing at once?
This code is just hypothetical to illustrate my question.
# X, y is the entire dataset.
x_train, y_train, x_test, y_test = train_test_split(X,y,stratify=y, test_size=.2)
# compared to
for x_bat, y_bat in stream_next_batch_from_file():
x_train, y_train, x_test, y_test = train_test_split(x_bat, y_bat, stratify=y_bat, test_size=.2)
# Append the splits to their respective files.
append_data(x_train, y_train, "train_set_filename")
append_data(x_test, y_test, "test_set_filename")
# etc.
scikit-learn dataset data
scikit-learn dataset data
asked Mar 27 at 21:57
Joey CarsonJoey Carson
1084
asked Mar 27 at 21:57
Joey CarsonJoey Carson
1084
asked Mar 27 at 21:57
Joey CarsonJoey Carson
1084
asked Mar 27 at 21:57
Joey CarsonJoey Carson
1084
asked Mar 27 at 21:57
Joey CarsonJoey Carson
1084
1084
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
$begingroup$
This should be fine. If the data is independent of its position in the list, this should give basically identical results.
If the data depends on order, then grabbing some data for training and testing from each batch will have a smoothing effect, making the training and test sets more representative than the results you would get with a raw train_test_split.
answered Mar 27 at 22:17
MegaTomMegaTom
1162
$endgroup$
add a comment |
Your Answer
StackExchange.ifUsing("editor", function ()
return StackExchange.using("mathjaxEditing", function ()
StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix)
StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["$", "$"], ["\\(","\\)"]]);
);
);
, "mathjax-editing");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "557"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f48112%2fcan-i-accurately-call-sklearn-model-selection-train-test-split-multiple-times-wh%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
$begingroup$
This should be fine. If the data is independent of its position in the list, this should give basically identical results.
If the data depends on order, then grabbing some data for training and testing from each batch will have a smoothing effect, making the training and test sets more representative than the results you would get with a raw train_test_split.
answered Mar 27 at 22:17
MegaTomMegaTom
1162
$endgroup$
add a comment |
$begingroup$
This should be fine. If the data is independent of its position in the list, this should give basically identical results.
If the data depends on order, then grabbing some data for training and testing from each batch will have a smoothing effect, making the training and test sets more representative than the results you would get with a raw train_test_split.
answered Mar 27 at 22:17
MegaTomMegaTom
1162
$endgroup$
add a comment |
$begingroup$
This should be fine. If the data is independent of its position in the list, this should give basically identical results.
If the data depends on order, then grabbing some data for training and testing from each batch will have a smoothing effect, making the training and test sets more representative than the results you would get with a raw train_test_split.
answered Mar 27 at 22:17
MegaTomMegaTom
1162
$endgroup$
This should be fine. If the data is independent of its position in the list, this should give basically identical results.
If the data depends on order, then grabbing some data for training and testing from each batch will have a smoothing effect, making the training and test sets more representative than the results you would get with a raw train_test_split.
answered Mar 27 at 22:17
MegaTomMegaTom
1162
answered Mar 27 at 22:17
MegaTomMegaTom
1162
answered Mar 27 at 22:17
MegaTomMegaTom
1162
answered Mar 27 at 22:17
MegaTomMegaTom
1162
1162
add a comment |
add a comment |
Thanks for contributing an answer to Data Science Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
Use MathJax to format equations. MathJax reference.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f48112%2fcan-i-accurately-call-sklearn-model-selection-train-test-split-multiple-times-wh%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
