Scikit learn train test split without mixing participants in trailsHow to split train/test in recommender systemsTraining Validation Testing set split for facial expression datasetConvolutional Neural Network not learning EEG dataFeatures standardization - Multilayer perceptronTraining score at parameter tuning lower than on hold out test set (RandomForestClassifier)Machine learning - 'train_test_split' function in scikit-learn: should I repeat it several times?Relationship between train and test errorHyper-parameter tuning when you don't have an access to the test dataOversampling before Cross-Validation, is it a problem?SciKit-Learn Decision Tree Overfitting

Has the laser at Magurele, Romania reached a tenth of the Sun's power?

Why is so much work done on numerical verification of the Riemann Hypothesis?

Is there any evidence that Cleopatra and Caesarion considered fleeing to India to escape the Romans?

How do you make your own symbol when Detexify fails?

Shouldn’t conservatives embrace universal basic income?

How to explain what's wrong with this application of the chain rule?

How to make money from a browser who sees 5 seconds into the future of any web page?

What is the English pronunciation of "pain au chocolat"?

What does "Scientists rise up against statistical significance" mean? (Comment in Nature)

Is there a nicer/politer/more positive alternative for "negates"?

Do we have to expect a queue for the shuttle from Watford Junction to Harry Potter Studio?

How would you translate "more" for use as an interface button?

"before" and "want" for the same systemd service?

Does "he squandered his car on drink" sound natural?

PTIJ: Why is Haman obsessed with Bose?

How much of a Devil Fruit must be consumed to gain the power?

Did the UK lift the requirement for registering SIM cards?

Quoting Keynes in a lecture

How do I fix the group tension caused by my character stealing and possibly killing without provocation?

Permission on Database

Has any country ever had 2 former presidents in jail simultaneously?

What is the highest possible scrabble score for placing a single tile

Doesn't the system of the Supreme Court oppose justice?

What does Apple's new App Store requirement mean



Scikit learn train test split without mixing participants in trails


How to split train/test in recommender systemsTraining Validation Testing set split for facial expression datasetConvolutional Neural Network not learning EEG dataFeatures standardization - Multilayer perceptronTraining score at parameter tuning lower than on hold out test set (RandomForestClassifier)Machine learning - 'train_test_split' function in scikit-learn: should I repeat it several times?Relationship between train and test errorHyper-parameter tuning when you don't have an access to the test dataOversampling before Cross-Validation, is it a problem?SciKit-Learn Decision Tree Overfitting













2












$begingroup$


I have dataset of trails. There are 90 participants that appear in 90 trails each. So 450 trails in total.



I'm looking to split my dataset so participants doesn't get mixed over training and test set. I want all trails of 25 participants in my training set and trails of the remaining 25 in the test set.



The reason is I want to test on totally unseen data to access overfitting. Is there a way of achieving this within scikit-learn?










share|improve this question









New contributor




Andrew is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.







$endgroup$











  • $begingroup$
    What do you mean by "splitting based on participant?"
    $endgroup$
    – Ethan
    Mar 18 at 17:27










  • $begingroup$
    @Ethan As in, I would like to split based on participant IDs and then classify exercises. For example, if I wanted to do a 50/50 train/test split, then I would have 25 participants data in the training, then test on the other 25 participants data.
    $endgroup$
    – Andrew
    Mar 18 at 20:23










  • $begingroup$
    I tried editing to clear up some of the confusion about what you are asking. Maybe someone will be able to answer now, but I'm pretty sure it would be easier to implement this functionality yourself.
    $endgroup$
    – Simon Larsson
    2 days ago















2












$begingroup$


I have dataset of trails. There are 90 participants that appear in 90 trails each. So 450 trails in total.



I'm looking to split my dataset so participants doesn't get mixed over training and test set. I want all trails of 25 participants in my training set and trails of the remaining 25 in the test set.



The reason is I want to test on totally unseen data to access overfitting. Is there a way of achieving this within scikit-learn?










share|improve this question









New contributor




Andrew is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.







$endgroup$











  • $begingroup$
    What do you mean by "splitting based on participant?"
    $endgroup$
    – Ethan
    Mar 18 at 17:27










  • $begingroup$
    @Ethan As in, I would like to split based on participant IDs and then classify exercises. For example, if I wanted to do a 50/50 train/test split, then I would have 25 participants data in the training, then test on the other 25 participants data.
    $endgroup$
    – Andrew
    Mar 18 at 20:23










  • $begingroup$
    I tried editing to clear up some of the confusion about what you are asking. Maybe someone will be able to answer now, but I'm pretty sure it would be easier to implement this functionality yourself.
    $endgroup$
    – Simon Larsson
    2 days ago













2












2








2


1



$begingroup$


I have dataset of trails. There are 90 participants that appear in 90 trails each. So 450 trails in total.



I'm looking to split my dataset so participants doesn't get mixed over training and test set. I want all trails of 25 participants in my training set and trails of the remaining 25 in the test set.



The reason is I want to test on totally unseen data to access overfitting. Is there a way of achieving this within scikit-learn?










share|improve this question









New contributor




Andrew is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.







$endgroup$




I have dataset of trails. There are 90 participants that appear in 90 trails each. So 450 trails in total.



I'm looking to split my dataset so participants doesn't get mixed over training and test set. I want all trails of 25 participants in my training set and trails of the remaining 25 in the test set.



The reason is I want to test on totally unseen data to access overfitting. Is there a way of achieving this within scikit-learn?







machine-learning






share|improve this question









New contributor




Andrew is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.











share|improve this question









New contributor




Andrew is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.









share|improve this question




share|improve this question








edited 2 days ago









Simon Larsson

51510




51510






New contributor




Andrew is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.









asked Mar 18 at 17:01









AndrewAndrew

112




112




New contributor




Andrew is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.





New contributor





Andrew is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.






Andrew is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.











  • $begingroup$
    What do you mean by "splitting based on participant?"
    $endgroup$
    – Ethan
    Mar 18 at 17:27










  • $begingroup$
    @Ethan As in, I would like to split based on participant IDs and then classify exercises. For example, if I wanted to do a 50/50 train/test split, then I would have 25 participants data in the training, then test on the other 25 participants data.
    $endgroup$
    – Andrew
    Mar 18 at 20:23










  • $begingroup$
    I tried editing to clear up some of the confusion about what you are asking. Maybe someone will be able to answer now, but I'm pretty sure it would be easier to implement this functionality yourself.
    $endgroup$
    – Simon Larsson
    2 days ago
















  • $begingroup$
    What do you mean by "splitting based on participant?"
    $endgroup$
    – Ethan
    Mar 18 at 17:27










  • $begingroup$
    @Ethan As in, I would like to split based on participant IDs and then classify exercises. For example, if I wanted to do a 50/50 train/test split, then I would have 25 participants data in the training, then test on the other 25 participants data.
    $endgroup$
    – Andrew
    Mar 18 at 20:23










  • $begingroup$
    I tried editing to clear up some of the confusion about what you are asking. Maybe someone will be able to answer now, but I'm pretty sure it would be easier to implement this functionality yourself.
    $endgroup$
    – Simon Larsson
    2 days ago















$begingroup$
What do you mean by "splitting based on participant?"
$endgroup$
– Ethan
Mar 18 at 17:27




$begingroup$
What do you mean by "splitting based on participant?"
$endgroup$
– Ethan
Mar 18 at 17:27












$begingroup$
@Ethan As in, I would like to split based on participant IDs and then classify exercises. For example, if I wanted to do a 50/50 train/test split, then I would have 25 participants data in the training, then test on the other 25 participants data.
$endgroup$
– Andrew
Mar 18 at 20:23




$begingroup$
@Ethan As in, I would like to split based on participant IDs and then classify exercises. For example, if I wanted to do a 50/50 train/test split, then I would have 25 participants data in the training, then test on the other 25 participants data.
$endgroup$
– Andrew
Mar 18 at 20:23












$begingroup$
I tried editing to clear up some of the confusion about what you are asking. Maybe someone will be able to answer now, but I'm pretty sure it would be easier to implement this functionality yourself.
$endgroup$
– Simon Larsson
2 days ago




$begingroup$
I tried editing to clear up some of the confusion about what you are asking. Maybe someone will be able to answer now, but I'm pretty sure it would be easier to implement this functionality yourself.
$endgroup$
– Simon Larsson
2 days ago










3 Answers
3






active

oldest

votes


















2












$begingroup$

You can use one of scikit-learn's options for grouped data. In particular, GroupKFold should do the trick: something like



from sklearn.model_selection import GroupKFold
group_kfold = GroupKFold(n_splits=2)
group_kfold.get_n_splits(X, y, groups)


where groups is an array of group indices.






share|improve this answer









$endgroup$




















    1












    $begingroup$

    To split data into Training and Test sets in scikit-learn you can use the train_test_split function.



    To implement this:



    from sklearn.model_selection import train_test_split

    sklearn.model_selection.train_test_split()


    Then pass through the arguments that you would like to set as parameters. Some parameters of note you may like to tune are the sizes of the training and test sets as well as setting if the splits are shuffled or not.



    You can read about this further on its scikit-learn documentation page here.






    share|improve this answer









    $endgroup$












    • $begingroup$
      Yes, I am using that function. I don't think stratify is the correct parameter since all that is doing it looking to make sure proportions stay the same. I just want to split participant data as being used for training only or testing only.
      $endgroup$
      – Andrew
      Mar 18 at 22:09










    • $begingroup$
      train_test_split will by default, do a 75% training 25% testing split. Is this what you want to change? I am not sure I understand your question.
      $endgroup$
      – Ethan
      Mar 19 at 0:01











    • $begingroup$
      No, I want to split based on participants. Currently it randomly splits so all 50 participants are very likely going to be included in the training and testing portions.
      $endgroup$
      – Andrew
      2 days ago










    • $begingroup$
      I want to do a 50% train split with the additional parameter that participants are mutually excluded to either the train or test split. So there are 25 participants worth of data on the train side and 25 participants worth of data on the test side. Basically, is there an easy way to randomize 50 participants into two groups? It seems like the answer is no and requires a custom script.
      $endgroup$
      – Andrew
      2 days ago










    • $begingroup$
      What exactly do you mean by very likely to be included? If you have 50 samples, and pass them through train_test_split, all 50 samples will be included in either the training set or the test set. To your second comment, everything is mutually exclusive. No sample will be included in both the training and the test set.
      $endgroup$
      – Ethan
      2 days ago


















    0












    $begingroup$

    id_split = random.sample(range(50),25)
    id_split.sort()

    blinddata_train = pd.DataFrame()
    blinddata_test = pd.DataFrame()
    blindtarget_train = pd.DataFrame()
    blindtarget_test = pd.DataFrame()

    for i in range(50):
    if i in train_split:
    # print("Training: ", (i+1))
    blinddata_train=blinddata_train.append(data[(90*i):(90*i+89)])
    blindtarget_train=blindtarget_train.append(target[(90*i):(90*i+89)])
    else:
    # print("Testing: ", (i+1))
    blinddata_test=blinddata_test.append(data[(90*i):(90*i+89)])
    blinddtarget_test=blindtarget_test.append(target[(90*i):(90*i+89)])



    I just wrote a quick blurb to randomly select 25 participants out of the 50 participants. The data was already sorted to have the 90 trials from each participant consecutively (90 trials from ID1 followed by 90 trials from ID2, etc.). So I just appended to either train or test depending upon whether it was in the test or train split.






    share|improve this answer










    New contributor




    Andrew is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
    Check out our Code of Conduct.






    $endgroup$












      Your Answer





      StackExchange.ifUsing("editor", function ()
      return StackExchange.using("mathjaxEditing", function ()
      StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix)
      StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["$", "$"], ["\\(","\\)"]]);
      );
      );
      , "mathjax-editing");

      StackExchange.ready(function()
      var channelOptions =
      tags: "".split(" "),
      id: "557"
      ;
      initTagRenderer("".split(" "), "".split(" "), channelOptions);

      StackExchange.using("externalEditor", function()
      // Have to fire editor after snippets, if snippets enabled
      if (StackExchange.settings.snippets.snippetsEnabled)
      StackExchange.using("snippets", function()
      createEditor();
      );

      else
      createEditor();

      );

      function createEditor()
      StackExchange.prepareEditor(
      heartbeatType: 'answer',
      autoActivateHeartbeat: false,
      convertImagesToLinks: false,
      noModals: true,
      showLowRepImageUploadWarning: true,
      reputationToPostImages: null,
      bindNavPrevention: true,
      postfix: "",
      imageUploader:
      brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
      contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
      allowUrls: true
      ,
      onDemand: true,
      discardSelector: ".discard-answer"
      ,immediatelyShowMarkdownHelp:true
      );



      );






      Andrew is a new contributor. Be nice, and check out our Code of Conduct.









      draft saved

      draft discarded


















      StackExchange.ready(
      function ()
      StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f47544%2fscikit-learn-train-test-split-without-mixing-participants-in-trails%23new-answer', 'question_page');

      );

      Post as a guest















      Required, but never shown

























      3 Answers
      3






      active

      oldest

      votes








      3 Answers
      3






      active

      oldest

      votes









      active

      oldest

      votes






      active

      oldest

      votes









      2












      $begingroup$

      You can use one of scikit-learn's options for grouped data. In particular, GroupKFold should do the trick: something like



      from sklearn.model_selection import GroupKFold
      group_kfold = GroupKFold(n_splits=2)
      group_kfold.get_n_splits(X, y, groups)


      where groups is an array of group indices.






      share|improve this answer









      $endgroup$

















        2












        $begingroup$

        You can use one of scikit-learn's options for grouped data. In particular, GroupKFold should do the trick: something like



        from sklearn.model_selection import GroupKFold
        group_kfold = GroupKFold(n_splits=2)
        group_kfold.get_n_splits(X, y, groups)


        where groups is an array of group indices.






        share|improve this answer









        $endgroup$















          2












          2








          2





          $begingroup$

          You can use one of scikit-learn's options for grouped data. In particular, GroupKFold should do the trick: something like



          from sklearn.model_selection import GroupKFold
          group_kfold = GroupKFold(n_splits=2)
          group_kfold.get_n_splits(X, y, groups)


          where groups is an array of group indices.






          share|improve this answer









          $endgroup$



          You can use one of scikit-learn's options for grouped data. In particular, GroupKFold should do the trick: something like



          from sklearn.model_selection import GroupKFold
          group_kfold = GroupKFold(n_splits=2)
          group_kfold.get_n_splits(X, y, groups)


          where groups is an array of group indices.







          share|improve this answer












          share|improve this answer



          share|improve this answer










          answered yesterday









          oW_oW_

          3,256730




          3,256730





















              1












              $begingroup$

              To split data into Training and Test sets in scikit-learn you can use the train_test_split function.



              To implement this:



              from sklearn.model_selection import train_test_split

              sklearn.model_selection.train_test_split()


              Then pass through the arguments that you would like to set as parameters. Some parameters of note you may like to tune are the sizes of the training and test sets as well as setting if the splits are shuffled or not.



              You can read about this further on its scikit-learn documentation page here.






              share|improve this answer









              $endgroup$












              • $begingroup$
                Yes, I am using that function. I don't think stratify is the correct parameter since all that is doing it looking to make sure proportions stay the same. I just want to split participant data as being used for training only or testing only.
                $endgroup$
                – Andrew
                Mar 18 at 22:09










              • $begingroup$
                train_test_split will by default, do a 75% training 25% testing split. Is this what you want to change? I am not sure I understand your question.
                $endgroup$
                – Ethan
                Mar 19 at 0:01











              • $begingroup$
                No, I want to split based on participants. Currently it randomly splits so all 50 participants are very likely going to be included in the training and testing portions.
                $endgroup$
                – Andrew
                2 days ago










              • $begingroup$
                I want to do a 50% train split with the additional parameter that participants are mutually excluded to either the train or test split. So there are 25 participants worth of data on the train side and 25 participants worth of data on the test side. Basically, is there an easy way to randomize 50 participants into two groups? It seems like the answer is no and requires a custom script.
                $endgroup$
                – Andrew
                2 days ago










              • $begingroup$
                What exactly do you mean by very likely to be included? If you have 50 samples, and pass them through train_test_split, all 50 samples will be included in either the training set or the test set. To your second comment, everything is mutually exclusive. No sample will be included in both the training and the test set.
                $endgroup$
                – Ethan
                2 days ago















              1












              $begingroup$

              To split data into Training and Test sets in scikit-learn you can use the train_test_split function.



              To implement this:



              from sklearn.model_selection import train_test_split

              sklearn.model_selection.train_test_split()


              Then pass through the arguments that you would like to set as parameters. Some parameters of note you may like to tune are the sizes of the training and test sets as well as setting if the splits are shuffled or not.



              You can read about this further on its scikit-learn documentation page here.






              share|improve this answer









              $endgroup$












              • $begingroup$
                Yes, I am using that function. I don't think stratify is the correct parameter since all that is doing it looking to make sure proportions stay the same. I just want to split participant data as being used for training only or testing only.
                $endgroup$
                – Andrew
                Mar 18 at 22:09










              • $begingroup$
                train_test_split will by default, do a 75% training 25% testing split. Is this what you want to change? I am not sure I understand your question.
                $endgroup$
                – Ethan
                Mar 19 at 0:01











              • $begingroup$
                No, I want to split based on participants. Currently it randomly splits so all 50 participants are very likely going to be included in the training and testing portions.
                $endgroup$
                – Andrew
                2 days ago










              • $begingroup$
                I want to do a 50% train split with the additional parameter that participants are mutually excluded to either the train or test split. So there are 25 participants worth of data on the train side and 25 participants worth of data on the test side. Basically, is there an easy way to randomize 50 participants into two groups? It seems like the answer is no and requires a custom script.
                $endgroup$
                – Andrew
                2 days ago










              • $begingroup$
                What exactly do you mean by very likely to be included? If you have 50 samples, and pass them through train_test_split, all 50 samples will be included in either the training set or the test set. To your second comment, everything is mutually exclusive. No sample will be included in both the training and the test set.
                $endgroup$
                – Ethan
                2 days ago













              1












              1








              1





              $begingroup$

              To split data into Training and Test sets in scikit-learn you can use the train_test_split function.



              To implement this:



              from sklearn.model_selection import train_test_split

              sklearn.model_selection.train_test_split()


              Then pass through the arguments that you would like to set as parameters. Some parameters of note you may like to tune are the sizes of the training and test sets as well as setting if the splits are shuffled or not.



              You can read about this further on its scikit-learn documentation page here.






              share|improve this answer









              $endgroup$



              To split data into Training and Test sets in scikit-learn you can use the train_test_split function.



              To implement this:



              from sklearn.model_selection import train_test_split

              sklearn.model_selection.train_test_split()


              Then pass through the arguments that you would like to set as parameters. Some parameters of note you may like to tune are the sizes of the training and test sets as well as setting if the splits are shuffled or not.



              You can read about this further on its scikit-learn documentation page here.







              share|improve this answer












              share|improve this answer



              share|improve this answer










              answered Mar 18 at 17:32









              EthanEthan

              576223




              576223











              • $begingroup$
                Yes, I am using that function. I don't think stratify is the correct parameter since all that is doing it looking to make sure proportions stay the same. I just want to split participant data as being used for training only or testing only.
                $endgroup$
                – Andrew
                Mar 18 at 22:09










              • $begingroup$
                train_test_split will by default, do a 75% training 25% testing split. Is this what you want to change? I am not sure I understand your question.
                $endgroup$
                – Ethan
                Mar 19 at 0:01











              • $begingroup$
                No, I want to split based on participants. Currently it randomly splits so all 50 participants are very likely going to be included in the training and testing portions.
                $endgroup$
                – Andrew
                2 days ago










              • $begingroup$
                I want to do a 50% train split with the additional parameter that participants are mutually excluded to either the train or test split. So there are 25 participants worth of data on the train side and 25 participants worth of data on the test side. Basically, is there an easy way to randomize 50 participants into two groups? It seems like the answer is no and requires a custom script.
                $endgroup$
                – Andrew
                2 days ago










              • $begingroup$
                What exactly do you mean by very likely to be included? If you have 50 samples, and pass them through train_test_split, all 50 samples will be included in either the training set or the test set. To your second comment, everything is mutually exclusive. No sample will be included in both the training and the test set.
                $endgroup$
                – Ethan
                2 days ago
















              • $begingroup$
                Yes, I am using that function. I don't think stratify is the correct parameter since all that is doing it looking to make sure proportions stay the same. I just want to split participant data as being used for training only or testing only.
                $endgroup$
                – Andrew
                Mar 18 at 22:09










              • $begingroup$
                train_test_split will by default, do a 75% training 25% testing split. Is this what you want to change? I am not sure I understand your question.
                $endgroup$
                – Ethan
                Mar 19 at 0:01











              • $begingroup$
                No, I want to split based on participants. Currently it randomly splits so all 50 participants are very likely going to be included in the training and testing portions.
                $endgroup$
                – Andrew
                2 days ago










              • $begingroup$
                I want to do a 50% train split with the additional parameter that participants are mutually excluded to either the train or test split. So there are 25 participants worth of data on the train side and 25 participants worth of data on the test side. Basically, is there an easy way to randomize 50 participants into two groups? It seems like the answer is no and requires a custom script.
                $endgroup$
                – Andrew
                2 days ago










              • $begingroup$
                What exactly do you mean by very likely to be included? If you have 50 samples, and pass them through train_test_split, all 50 samples will be included in either the training set or the test set. To your second comment, everything is mutually exclusive. No sample will be included in both the training and the test set.
                $endgroup$
                – Ethan
                2 days ago















              $begingroup$
              Yes, I am using that function. I don't think stratify is the correct parameter since all that is doing it looking to make sure proportions stay the same. I just want to split participant data as being used for training only or testing only.
              $endgroup$
              – Andrew
              Mar 18 at 22:09




              $begingroup$
              Yes, I am using that function. I don't think stratify is the correct parameter since all that is doing it looking to make sure proportions stay the same. I just want to split participant data as being used for training only or testing only.
              $endgroup$
              – Andrew
              Mar 18 at 22:09












              $begingroup$
              train_test_split will by default, do a 75% training 25% testing split. Is this what you want to change? I am not sure I understand your question.
              $endgroup$
              – Ethan
              Mar 19 at 0:01





              $begingroup$
              train_test_split will by default, do a 75% training 25% testing split. Is this what you want to change? I am not sure I understand your question.
              $endgroup$
              – Ethan
              Mar 19 at 0:01













              $begingroup$
              No, I want to split based on participants. Currently it randomly splits so all 50 participants are very likely going to be included in the training and testing portions.
              $endgroup$
              – Andrew
              2 days ago




              $begingroup$
              No, I want to split based on participants. Currently it randomly splits so all 50 participants are very likely going to be included in the training and testing portions.
              $endgroup$
              – Andrew
              2 days ago












              $begingroup$
              I want to do a 50% train split with the additional parameter that participants are mutually excluded to either the train or test split. So there are 25 participants worth of data on the train side and 25 participants worth of data on the test side. Basically, is there an easy way to randomize 50 participants into two groups? It seems like the answer is no and requires a custom script.
              $endgroup$
              – Andrew
              2 days ago




              $begingroup$
              I want to do a 50% train split with the additional parameter that participants are mutually excluded to either the train or test split. So there are 25 participants worth of data on the train side and 25 participants worth of data on the test side. Basically, is there an easy way to randomize 50 participants into two groups? It seems like the answer is no and requires a custom script.
              $endgroup$
              – Andrew
              2 days ago












              $begingroup$
              What exactly do you mean by very likely to be included? If you have 50 samples, and pass them through train_test_split, all 50 samples will be included in either the training set or the test set. To your second comment, everything is mutually exclusive. No sample will be included in both the training and the test set.
              $endgroup$
              – Ethan
              2 days ago




              $begingroup$
              What exactly do you mean by very likely to be included? If you have 50 samples, and pass them through train_test_split, all 50 samples will be included in either the training set or the test set. To your second comment, everything is mutually exclusive. No sample will be included in both the training and the test set.
              $endgroup$
              – Ethan
              2 days ago











              0












              $begingroup$

              id_split = random.sample(range(50),25)
              id_split.sort()

              blinddata_train = pd.DataFrame()
              blinddata_test = pd.DataFrame()
              blindtarget_train = pd.DataFrame()
              blindtarget_test = pd.DataFrame()

              for i in range(50):
              if i in train_split:
              # print("Training: ", (i+1))
              blinddata_train=blinddata_train.append(data[(90*i):(90*i+89)])
              blindtarget_train=blindtarget_train.append(target[(90*i):(90*i+89)])
              else:
              # print("Testing: ", (i+1))
              blinddata_test=blinddata_test.append(data[(90*i):(90*i+89)])
              blinddtarget_test=blindtarget_test.append(target[(90*i):(90*i+89)])



              I just wrote a quick blurb to randomly select 25 participants out of the 50 participants. The data was already sorted to have the 90 trials from each participant consecutively (90 trials from ID1 followed by 90 trials from ID2, etc.). So I just appended to either train or test depending upon whether it was in the test or train split.






              share|improve this answer










              New contributor




              Andrew is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
              Check out our Code of Conduct.






              $endgroup$

















                0












                $begingroup$

                id_split = random.sample(range(50),25)
                id_split.sort()

                blinddata_train = pd.DataFrame()
                blinddata_test = pd.DataFrame()
                blindtarget_train = pd.DataFrame()
                blindtarget_test = pd.DataFrame()

                for i in range(50):
                if i in train_split:
                # print("Training: ", (i+1))
                blinddata_train=blinddata_train.append(data[(90*i):(90*i+89)])
                blindtarget_train=blindtarget_train.append(target[(90*i):(90*i+89)])
                else:
                # print("Testing: ", (i+1))
                blinddata_test=blinddata_test.append(data[(90*i):(90*i+89)])
                blinddtarget_test=blindtarget_test.append(target[(90*i):(90*i+89)])



                I just wrote a quick blurb to randomly select 25 participants out of the 50 participants. The data was already sorted to have the 90 trials from each participant consecutively (90 trials from ID1 followed by 90 trials from ID2, etc.). So I just appended to either train or test depending upon whether it was in the test or train split.






                share|improve this answer










                New contributor




                Andrew is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
                Check out our Code of Conduct.






                $endgroup$















                  0












                  0








                  0





                  $begingroup$

                  id_split = random.sample(range(50),25)
                  id_split.sort()

                  blinddata_train = pd.DataFrame()
                  blinddata_test = pd.DataFrame()
                  blindtarget_train = pd.DataFrame()
                  blindtarget_test = pd.DataFrame()

                  for i in range(50):
                  if i in train_split:
                  # print("Training: ", (i+1))
                  blinddata_train=blinddata_train.append(data[(90*i):(90*i+89)])
                  blindtarget_train=blindtarget_train.append(target[(90*i):(90*i+89)])
                  else:
                  # print("Testing: ", (i+1))
                  blinddata_test=blinddata_test.append(data[(90*i):(90*i+89)])
                  blinddtarget_test=blindtarget_test.append(target[(90*i):(90*i+89)])



                  I just wrote a quick blurb to randomly select 25 participants out of the 50 participants. The data was already sorted to have the 90 trials from each participant consecutively (90 trials from ID1 followed by 90 trials from ID2, etc.). So I just appended to either train or test depending upon whether it was in the test or train split.






                  share|improve this answer










                  New contributor




                  Andrew is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
                  Check out our Code of Conduct.






                  $endgroup$



                  id_split = random.sample(range(50),25)
                  id_split.sort()

                  blinddata_train = pd.DataFrame()
                  blinddata_test = pd.DataFrame()
                  blindtarget_train = pd.DataFrame()
                  blindtarget_test = pd.DataFrame()

                  for i in range(50):
                  if i in train_split:
                  # print("Training: ", (i+1))
                  blinddata_train=blinddata_train.append(data[(90*i):(90*i+89)])
                  blindtarget_train=blindtarget_train.append(target[(90*i):(90*i+89)])
                  else:
                  # print("Testing: ", (i+1))
                  blinddata_test=blinddata_test.append(data[(90*i):(90*i+89)])
                  blinddtarget_test=blindtarget_test.append(target[(90*i):(90*i+89)])



                  I just wrote a quick blurb to randomly select 25 participants out of the 50 participants. The data was already sorted to have the 90 trials from each participant consecutively (90 trials from ID1 followed by 90 trials from ID2, etc.). So I just appended to either train or test depending upon whether it was in the test or train split.







                  share|improve this answer










                  New contributor




                  Andrew is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
                  Check out our Code of Conduct.









                  share|improve this answer



                  share|improve this answer








                  edited yesterday





















                  New contributor




                  Andrew is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
                  Check out our Code of Conduct.









                  answered yesterday









                  AndrewAndrew

                  112




                  112




                  New contributor




                  Andrew is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
                  Check out our Code of Conduct.





                  New contributor





                  Andrew is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
                  Check out our Code of Conduct.






                  Andrew is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
                  Check out our Code of Conduct.




















                      Andrew is a new contributor. Be nice, and check out our Code of Conduct.









                      draft saved

                      draft discarded


















                      Andrew is a new contributor. Be nice, and check out our Code of Conduct.












                      Andrew is a new contributor. Be nice, and check out our Code of Conduct.











                      Andrew is a new contributor. Be nice, and check out our Code of Conduct.














                      Thanks for contributing an answer to Data Science Stack Exchange!


                      • Please be sure to answer the question. Provide details and share your research!

                      But avoid


                      • Asking for help, clarification, or responding to other answers.

                      • Making statements based on opinion; back them up with references or personal experience.

                      Use MathJax to format equations. MathJax reference.


                      To learn more, see our tips on writing great answers.




                      draft saved


                      draft discarded














                      StackExchange.ready(
                      function ()
                      StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f47544%2fscikit-learn-train-test-split-without-mixing-participants-in-trails%23new-answer', 'question_page');

                      );

                      Post as a guest















                      Required, but never shown





















































                      Required, but never shown














                      Required, but never shown












                      Required, but never shown







                      Required, but never shown

































                      Required, but never shown














                      Required, but never shown












                      Required, but never shown







                      Required, but never shown







                      Popular posts from this blog

                      Adding axes to figuresAdding axes labels to LaTeX figuresLaTeX equivalent of ConTeXt buffersRotate a node but not its content: the case of the ellipse decorationHow to define the default vertical distance between nodes?TikZ scaling graphic and adjust node position and keep font sizeNumerical conditional within tikz keys?adding axes to shapesAlign axes across subfiguresAdding figures with a certain orderLine up nested tikz enviroments or how to get rid of themAdding axes labels to LaTeX figures

                      Luettelo Yhdysvaltain laivaston lentotukialuksista Lähteet | Navigointivalikko

                      Gary (muusikko) Sisällysluettelo Historia | Rockin' High | Lähteet | Aiheesta muualla | NavigointivalikkoInfobox OKTuomas "Gary" Keskinen Ancaran kitaristiksiProjekti Rockin' High