How could indestructible materials be used in power generation?

Why does ы have a soft sign in it?

Why is Collection not simply treated as Collection<?>

Python: return float 1.0 as int 1 but float 1.5 as float 1.5

Expand and Contract

What is the difference between 仮定 and 想定?

Is it inappropriate for a student to attend their mentor's dissertation defense?

A category-like structure without composition?

What method can I use to design a dungeon difficult enough that the PCs can't make it through without killing them?

How do I write bicross product symbols in latex?

Alternative to sending password over mail?

A reference to a well-known characterization of scattered compact spaces

Is "remove commented out code" correct English?

Why doesn't using multiple commands with a || or && conditional work?

What is the intuition behind short exact sequences of groups; in particular, what is the intuition behind group extensions?

Is there a way of "bevelling" a single vertex?

Twin primes whose sum is a cube

Why doesn't H₄O²⁺ exist?

Anagram holiday

What does it mean to describe someone as a butt steak?

Why "Having chlorophyll without photosynthesis is actually very dangerous" and "like living with a bomb"?

What is the most common color to indicate the input-field is disabled?

What do you call someone who asks many questions?

How to say in German "enjoying home comforts"

How to correctly apply the same data transformation , used on the training dataset , on real data in a webservice?

2

1

$begingroup$

Let's say I used minmaxscaler while creating my model.
Now, i'm loading that model via Pickle in a Flask app. Upon receiving a request containing a datapoint I would like to apply to it the same transformations that I applied to my training dataset before calling the predict() method. How do I transfer that set of transformations from one file to a webservice?

machine-learning data

share|improve this question

edited Mar 27 at 3:23

Ethan

671425

asked Mar 26 at 13:52

BlenzusBlenzus

14610

$endgroup$

add a comment | 

2

1

$begingroup$

Let's say I used minmaxscaler while creating my model.
Now, i'm loading that model via Pickle in a Flask app. Upon receiving a request containing a datapoint I would like to apply to it the same transformations that I applied to my training dataset before calling the predict() method. How do I transfer that set of transformations from one file to a webservice?

machine-learning data

share|improve this question

edited Mar 27 at 3:23

Ethan

671425

asked Mar 26 at 13:52

BlenzusBlenzus

14610

$endgroup$

add a comment | 

$begingroup$

Let's say I used minmaxscaler while creating my model.
Now, i'm loading that model via Pickle in a Flask app. Upon receiving a request containing a datapoint I would like to apply to it the same transformations that I applied to my training dataset before calling the predict() method. How do I transfer that set of transformations from one file to a webservice?

machine-learning data

share|improve this question

edited Mar 27 at 3:23

Ethan

671425

asked Mar 26 at 13:52

BlenzusBlenzus

14610

$endgroup$

share|improve this question

edited Mar 27 at 3:23

Ethan

671425

asked Mar 26 at 13:52

BlenzusBlenzus

14610

share|improve this question

edited Mar 27 at 3:23

Ethan

671425

asked Mar 26 at 13:52

BlenzusBlenzus

14610

edited Mar 27 at 3:23

Ethan

671425

edited Mar 27 at 3:23

Ethan

671425

asked Mar 26 at 13:52

BlenzusBlenzus

14610

asked Mar 26 at 13:52

BlenzusBlenzus

14610

2 Answers
2

active

oldest

votes

2

$begingroup$

Rather than storing and loading many files, create a Scikit-learn transformation pipeline with all of your transformations, and then save that as a pickle or joblib file.

from sklearn.pipeline import Pipeline
from sklearn.externals import joblib

pipeline = Pipeline([
 ('normalization', MinMaxScaler()),
 ('classifier', RandomForestClassifier())
 ])

joblib.dump(pipeline, 'transform_predict.joblib')

You can then just load one transformation pipeline and call fit_transform to transform the input data and get predictions for it:

 pipeline = load('transform_predict.joblib') 
 predictions = pipeline.predict(new_data)

share|improve this answer

edited Mar 26 at 14:45

answered Mar 26 at 14:40

Dan CarterDan Carter

8351218

$endgroup$

• 
  
  
  

  
  1
  

  
  
  
  
  
  

  
  $begingroup$
  Thanks, this is what i was looking for
  $endgroup$
  – Blenzus
  Mar 26 at 14:43
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  Does this apply to dummy variables?
  $endgroup$
  – Blenzus
  Mar 26 at 14:55
  

  
  
  
  


• 
  
  
  

  
  1
  

  
  
  
  
  
  

  
  $begingroup$
  If you're using scikit-learn's OneHotEncoder then yes. Any scikit learn 'transformer' can be used with a pipeline, so anything that implements the TransformerMixin and BaseEstimator: github.com/scikit-learn/scikit-learn/blob/7b136e9/sklearn/… This also means you can create your own custom 'transformers' to add to a pipeline, by implementing these in the same way.
  $endgroup$
  – Dan Carter
  Mar 26 at 15:34
  

  
  
  
  

add a comment | 

1

$begingroup$

You need to save minmaxscaler (along with model). In Flask app, you can :

1. Load scaler from file


2. Use this instance of scaler for scaling input values

#While training

from sklearn.externals import joblib
scaler_filename = "saved_scaler"
joblib.dump(scaler, scaler_filename)

In Flask App

scaler_filename = "saved_scaler" 
scaler = joblib.load(scaler_filename)

share|improve this answer

answered Mar 26 at 14:08

Shamit VermaShamit Verma

1,3191214

$endgroup$

• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  Do i need to do this for every normalization library i use? i'll be loading many files into the memory, isn't there way to load something that contains every step of the data transformations?
  $endgroup$
  – Blenzus
  Mar 26 at 14:17
  

  
  
  
  


• 
  
  
  

  
  1
  

  
  
  
  
  
  

  
  $begingroup$
  You can save and load all scalers at the same time. Example : stackoverflow.com/questions/33497314/…
  $endgroup$
  – Shamit Verma
  Mar 26 at 14:33
  

  
  
  
  

add a comment | 

Your Answer

StackExchange.ifUsing("editor", function ()
return StackExchange.using("mathjaxEditing", function ()
StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix)
StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["$", "$"], ["\\(","\\)"]]);
);
);
, "mathjax-editing");

StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "557"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);

else
createEditor();

);

function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);



);

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);

Sign up using Google

Sign up using Facebook

Sign up using Email and Password

Post as a guest

Name

Email

Required, but never shown

StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f48026%2fhow-to-correctly-apply-the-same-data-transformation-used-on-the-training-datas%23new-answer', 'question_page');

);

Post as a guest

Name

Email

Required, but never shown

2

$begingroup$

Rather than storing and loading many files, create a Scikit-learn transformation pipeline with all of your transformations, and then save that as a pickle or joblib file.

from sklearn.pipeline import Pipeline
from sklearn.externals import joblib

pipeline = Pipeline([
 ('normalization', MinMaxScaler()),
 ('classifier', RandomForestClassifier())
 ])

joblib.dump(pipeline, 'transform_predict.joblib')

You can then just load one transformation pipeline and call fit_transform to transform the input data and get predictions for it:

 pipeline = load('transform_predict.joblib') 
 predictions = pipeline.predict(new_data)

share|improve this answer

edited Mar 26 at 14:45

answered Mar 26 at 14:40

Dan CarterDan Carter

8351218

$endgroup$

• 
  
  
  

  
  1
  

  
  
  
  
  
  

  
  $begingroup$
  Thanks, this is what i was looking for
  $endgroup$
  – Blenzus
  Mar 26 at 14:43
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  Does this apply to dummy variables?
  $endgroup$
  – Blenzus
  Mar 26 at 14:55
  

  
  
  
  


• 
  
  
  

  
  1
  

  
  
  
  
  
  

  
  $begingroup$
  If you're using scikit-learn's OneHotEncoder then yes. Any scikit learn 'transformer' can be used with a pipeline, so anything that implements the TransformerMixin and BaseEstimator: github.com/scikit-learn/scikit-learn/blob/7b136e9/sklearn/… This also means you can create your own custom 'transformers' to add to a pipeline, by implementing these in the same way.
  $endgroup$
  – Dan Carter
  Mar 26 at 15:34
  

  
  
  
  

add a comment | 

2

$begingroup$

Rather than storing and loading many files, create a Scikit-learn transformation pipeline with all of your transformations, and then save that as a pickle or joblib file.

from sklearn.pipeline import Pipeline
from sklearn.externals import joblib

pipeline = Pipeline([
 ('normalization', MinMaxScaler()),
 ('classifier', RandomForestClassifier())
 ])

joblib.dump(pipeline, 'transform_predict.joblib')

You can then just load one transformation pipeline and call fit_transform to transform the input data and get predictions for it:

 pipeline = load('transform_predict.joblib') 
 predictions = pipeline.predict(new_data)

share|improve this answer

edited Mar 26 at 14:45

answered Mar 26 at 14:40

Dan CarterDan Carter

8351218

$endgroup$

• 
  
  
  

  
  1
  

  
  
  
  
  
  

  
  $begingroup$
  Thanks, this is what i was looking for
  $endgroup$
  – Blenzus
  Mar 26 at 14:43
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  Does this apply to dummy variables?
  $endgroup$
  – Blenzus
  Mar 26 at 14:55
  

  
  
  
  


• 
  
  
  

  
  1
  

  
  
  
  
  
  

  
  $begingroup$
  If you're using scikit-learn's OneHotEncoder then yes. Any scikit learn 'transformer' can be used with a pipeline, so anything that implements the TransformerMixin and BaseEstimator: github.com/scikit-learn/scikit-learn/blob/7b136e9/sklearn/… This also means you can create your own custom 'transformers' to add to a pipeline, by implementing these in the same way.
  $endgroup$
  – Dan Carter
  Mar 26 at 15:34
  

  
  
  
  

add a comment | 

$begingroup$

Rather than storing and loading many files, create a Scikit-learn transformation pipeline with all of your transformations, and then save that as a pickle or joblib file.

from sklearn.pipeline import Pipeline
from sklearn.externals import joblib

pipeline = Pipeline([
 ('normalization', MinMaxScaler()),
 ('classifier', RandomForestClassifier())
 ])

joblib.dump(pipeline, 'transform_predict.joblib')

You can then just load one transformation pipeline and call fit_transform to transform the input data and get predictions for it:

 pipeline = load('transform_predict.joblib') 
 predictions = pipeline.predict(new_data)

share|improve this answer

edited Mar 26 at 14:45

answered Mar 26 at 14:40

Dan CarterDan Carter

8351218

$endgroup$

from sklearn.pipeline import Pipeline
from sklearn.externals import joblib

pipeline = Pipeline([
 ('normalization', MinMaxScaler()),
 ('classifier', RandomForestClassifier())
 ])

joblib.dump(pipeline, 'transform_predict.joblib')

 pipeline = load('transform_predict.joblib') 
 predictions = pipeline.predict(new_data)

share|improve this answer

edited Mar 26 at 14:45

answered Mar 26 at 14:40

Dan CarterDan Carter

8351218

answered Mar 26 at 14:40

Dan CarterDan Carter

8351218

answered Mar 26 at 14:40

Dan CarterDan Carter

8351218

1

$begingroup$

You need to save minmaxscaler (along with model). In Flask app, you can :

1. Load scaler from file


2. Use this instance of scaler for scaling input values

#While training

from sklearn.externals import joblib
scaler_filename = "saved_scaler"
joblib.dump(scaler, scaler_filename)

In Flask App

scaler_filename = "saved_scaler" 
scaler = joblib.load(scaler_filename)

share|improve this answer

answered Mar 26 at 14:08

Shamit VermaShamit Verma

1,3191214

$endgroup$

• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  Do i need to do this for every normalization library i use? i'll be loading many files into the memory, isn't there way to load something that contains every step of the data transformations?
  $endgroup$
  – Blenzus
  Mar 26 at 14:17
  

  
  
  
  


• 
  
  
  

  
  1
  

  
  
  
  
  
  

  
  $begingroup$
  You can save and load all scalers at the same time. Example : stackoverflow.com/questions/33497314/…
  $endgroup$
  – Shamit Verma
  Mar 26 at 14:33
  

  
  
  
  

add a comment | 

1

$begingroup$

You need to save minmaxscaler (along with model). In Flask app, you can :

1. Load scaler from file


2. Use this instance of scaler for scaling input values

#While training

from sklearn.externals import joblib
scaler_filename = "saved_scaler"
joblib.dump(scaler, scaler_filename)

In Flask App

scaler_filename = "saved_scaler" 
scaler = joblib.load(scaler_filename)

share|improve this answer

answered Mar 26 at 14:08

Shamit VermaShamit Verma

1,3191214

$endgroup$

• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  Do i need to do this for every normalization library i use? i'll be loading many files into the memory, isn't there way to load something that contains every step of the data transformations?
  $endgroup$
  – Blenzus
  Mar 26 at 14:17
  

  
  
  
  


• 
  
  
  

  
  1
  

  
  
  
  
  
  

  
  $begingroup$
  You can save and load all scalers at the same time. Example : stackoverflow.com/questions/33497314/…
  $endgroup$
  – Shamit Verma
  Mar 26 at 14:33
  

  
  
  
  

add a comment | 

$begingroup$

You need to save minmaxscaler (along with model). In Flask app, you can :

1. Load scaler from file


2. Use this instance of scaler for scaling input values

#While training

from sklearn.externals import joblib
scaler_filename = "saved_scaler"
joblib.dump(scaler, scaler_filename)

In Flask App

scaler_filename = "saved_scaler" 
scaler = joblib.load(scaler_filename)

share|improve this answer

answered Mar 26 at 14:08

Shamit VermaShamit Verma

1,3191214

$endgroup$

share|improve this answer

answered Mar 26 at 14:08

Shamit VermaShamit Verma

1,3191214

answered Mar 26 at 14:08

Shamit VermaShamit Verma

1,3191214

answered Mar 26 at 14:08

Shamit VermaShamit Verma

1,3191214