sklearn and pandas in AWS Lambda The Next CEO of Stack Overflow2019 Community Moderator ElectionStruggling to integrate sklearn and pandas in simple Kaggle taskDownloading a large dataset on the web directly into AWS S3AWS machine learning prediction schema problemsSklearn and PCA. Why is max n_row == max n_components?AWS model performance evaluation interpretationUsing AWS ML to recommend productsSyntax error in pandas dataframe slice in my sklearn data prep?Predicting contract churn/cancellation: Great model results does not work in the real worldmaximum dimensionality of AWS Machine LearningDifference between AWS amazonei_tensorflow and tensorflow conda environments
What's the best way to handle refactoring a big file?
Is it possible to search for a directory/file combination?
How to get the end in algorithm2e
Won the lottery - how do I keep the money?
Why do we use the plural of movies in this phrase "We went to the movies last night."?
Display a text message if the shortcode is not found?
Customer Requests (Sometimes) Drive Me Bonkers!
Make solar eclipses exceedingly rare, but still have new moons
Why does standard notation not preserve intervals (visually)
If Nick Fury and Coulson already knew about aliens (Kree and Skrull) why did they wait until Thor's appearance to start making weapons?
Does soap repel water?
Is wanting to ask what to write an indication that you need to change your story?
How long to clear the 'suck zone' of a turbofan after start is initiated?
Where do students learn to solve polynomial equations these days?
Why am I allowed to create multiple unique pointers from a single object?
Arranging cats and dogs - what is wrong with my approach
If the heap is zero-initialized for security, then why is the stack merely uninitialized?
Rotate a column
Why do airplanes bank sharply to the right after air-to-air refueling?
Would a completely good Muggle be able to use a wand?
Unreliable Magic - Is it worth it?
What happens if you roll doubles 3 times then land on "Go to jail?"
How to get from Geneva Airport to Metabief?
Flying from Cape Town to England and return to another province
sklearn and pandas in AWS Lambda
The Next CEO of Stack Overflow2019 Community Moderator ElectionStruggling to integrate sklearn and pandas in simple Kaggle taskDownloading a large dataset on the web directly into AWS S3AWS machine learning prediction schema problemsSklearn and PCA. Why is max n_row == max n_components?AWS model performance evaluation interpretationUsing AWS ML to recommend productsSyntax error in pandas dataframe slice in my sklearn data prep?Predicting contract churn/cancellation: Great model results does not work in the real worldmaximum dimensionality of AWS Machine LearningDifference between AWS amazonei_tensorflow and tensorflow conda environments
$begingroup$
I try to explain my problem the clearest as possible:
I made a front end with which I want make REST calls to an AWS Lambda interfaced with AWS API Gateway.
I dumped my model as a pkl file (and so my encoders) which I firstly trained locally. I stored these files in a S3 bucket.
The problem is that I cannot import libraries such as pandas and sklearn to make model predictions as the lambda console is unable to find them.
Do you have any solutions to solve this issue?
scikit-learn pandas aws aws-lambda
asked Mar 25 at 16:36
3nomis3nomis
19710
$endgroup$
add a comment |
$begingroup$
I try to explain my problem the clearest as possible:
I made a front end with which I want make REST calls to an AWS Lambda interfaced with AWS API Gateway.
I dumped my model as a pkl file (and so my encoders) which I firstly trained locally. I stored these files in a S3 bucket.
The problem is that I cannot import libraries such as pandas and sklearn to make model predictions as the lambda console is unable to find them.
Do you have any solutions to solve this issue?
scikit-learn pandas aws aws-lambda
asked Mar 25 at 16:36
3nomis3nomis
19710
$endgroup$
add a comment |
$begingroup$
I try to explain my problem the clearest as possible:
I made a front end with which I want make REST calls to an AWS Lambda interfaced with AWS API Gateway.
I dumped my model as a pkl file (and so my encoders) which I firstly trained locally. I stored these files in a S3 bucket.
The problem is that I cannot import libraries such as pandas and sklearn to make model predictions as the lambda console is unable to find them.
Do you have any solutions to solve this issue?
scikit-learn pandas aws aws-lambda
asked Mar 25 at 16:36
3nomis3nomis
19710
$endgroup$
I try to explain my problem the clearest as possible:
I made a front end with which I want make REST calls to an AWS Lambda interfaced with AWS API Gateway.
I dumped my model as a pkl file (and so my encoders) which I firstly trained locally. I stored these files in a S3 bucket.
The problem is that I cannot import libraries such as pandas and sklearn to make model predictions as the lambda console is unable to find them.
Do you have any solutions to solve this issue?
scikit-learn pandas aws aws-lambda
scikit-learn pandas aws aws-lambda
asked Mar 25 at 16:36
3nomis3nomis
19710
asked Mar 25 at 16:36
3nomis3nomis
19710
asked Mar 25 at 16:36
3nomis3nomis
19710
asked Mar 25 at 16:36
3nomis3nomis
19710
asked Mar 25 at 16:36
3nomis3nomis
19710
19710
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
$begingroup$
You need to create a deployment package which includes the packages you want to use in Lambda (sklearn and pandas).
You can then either upload that deployment package to S3 and import it in the Lambda function, or upload it within the Lambda function itself.
The Lambda function code will have to be written outside of AWS Lambda and be included in the deployment package. Here's a guide on how to do it.
answered Mar 25 at 16:43
Dan CarterDan Carter
8101218
$endgroup$
$begingroup$
i tried this solution, however when I reference the zip on s3 it shows a dimension exceeded. Didi you face the same issue or am I doing anything wrong?
$endgroup$
– 3nomis
Mar 26 at 8:53
$begingroup$
How big is your deployment package uncompressed?
$endgroup$
– Dan Carter
Mar 26 at 9:44
1
$begingroup$
Yes, you need to create the package in a Linux environment.
$endgroup$
– Dan Carter
yesterday
1
$begingroup$
I'd guess any Linux OS would be okay, but not sure on that one. I ssh'd into an Amazon EC2 instance and created the package there...
$endgroup$
– Dan Carter
yesterday
1
$begingroup$
Yes, numpy, pandas, sklearn. It will work :) Just make sure you only add the packages you need, obviously with lambda you pay for memory used per second, package size may affect that.
$endgroup$
– Dan Carter
yesterday
|
show 3 more comments
Your Answer
StackExchange.ifUsing("editor", function ()
return StackExchange.using("mathjaxEditing", function ()
StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix)
StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["$", "$"], ["\\(","\\)"]]);
);
);
, "mathjax-editing");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "557"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f47955%2fsklearn-and-pandas-in-aws-lambda%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
$begingroup$
You need to create a deployment package which includes the packages you want to use in Lambda (sklearn and pandas).
You can then either upload that deployment package to S3 and import it in the Lambda function, or upload it within the Lambda function itself.
The Lambda function code will have to be written outside of AWS Lambda and be included in the deployment package. Here's a guide on how to do it.
answered Mar 25 at 16:43
Dan CarterDan Carter
8101218
$endgroup$
$begingroup$
i tried this solution, however when I reference the zip on s3 it shows a dimension exceeded. Didi you face the same issue or am I doing anything wrong?
$endgroup$
– 3nomis
Mar 26 at 8:53
$begingroup$
How big is your deployment package uncompressed?
$endgroup$
– Dan Carter
Mar 26 at 9:44
1
$begingroup$
Yes, you need to create the package in a Linux environment.
$endgroup$
– Dan Carter
yesterday
1
$begingroup$
I'd guess any Linux OS would be okay, but not sure on that one. I ssh'd into an Amazon EC2 instance and created the package there...
$endgroup$
– Dan Carter
yesterday
1
$begingroup$
Yes, numpy, pandas, sklearn. It will work :) Just make sure you only add the packages you need, obviously with lambda you pay for memory used per second, package size may affect that.
$endgroup$
– Dan Carter
yesterday
|
show 3 more comments
$begingroup$
You need to create a deployment package which includes the packages you want to use in Lambda (sklearn and pandas).
You can then either upload that deployment package to S3 and import it in the Lambda function, or upload it within the Lambda function itself.
The Lambda function code will have to be written outside of AWS Lambda and be included in the deployment package. Here's a guide on how to do it.
answered Mar 25 at 16:43
Dan CarterDan Carter
8101218
$endgroup$
$begingroup$
i tried this solution, however when I reference the zip on s3 it shows a dimension exceeded. Didi you face the same issue or am I doing anything wrong?
$endgroup$
– 3nomis
Mar 26 at 8:53
$begingroup$
How big is your deployment package uncompressed?
$endgroup$
– Dan Carter
Mar 26 at 9:44
1
$begingroup$
Yes, you need to create the package in a Linux environment.
$endgroup$
– Dan Carter
yesterday
1
$begingroup$
I'd guess any Linux OS would be okay, but not sure on that one. I ssh'd into an Amazon EC2 instance and created the package there...
$endgroup$
– Dan Carter
yesterday
1
$begingroup$
Yes, numpy, pandas, sklearn. It will work :) Just make sure you only add the packages you need, obviously with lambda you pay for memory used per second, package size may affect that.
$endgroup$
– Dan Carter
yesterday
|
show 3 more comments
$begingroup$
You need to create a deployment package which includes the packages you want to use in Lambda (sklearn and pandas).
You can then either upload that deployment package to S3 and import it in the Lambda function, or upload it within the Lambda function itself.
The Lambda function code will have to be written outside of AWS Lambda and be included in the deployment package. Here's a guide on how to do it.
answered Mar 25 at 16:43
Dan CarterDan Carter
8101218
$endgroup$
You need to create a deployment package which includes the packages you want to use in Lambda (sklearn and pandas).
You can then either upload that deployment package to S3 and import it in the Lambda function, or upload it within the Lambda function itself.
The Lambda function code will have to be written outside of AWS Lambda and be included in the deployment package. Here's a guide on how to do it.
answered Mar 25 at 16:43
Dan CarterDan Carter
8101218
answered Mar 25 at 16:43
Dan CarterDan Carter
8101218
answered Mar 25 at 16:43
Dan CarterDan Carter
8101218
answered Mar 25 at 16:43
Dan CarterDan Carter
8101218
8101218
$begingroup$
i tried this solution, however when I reference the zip on s3 it shows a dimension exceeded. Didi you face the same issue or am I doing anything wrong?
$endgroup$
– 3nomis
Mar 26 at 8:53
$begingroup$
How big is your deployment package uncompressed?
$endgroup$
– Dan Carter
Mar 26 at 9:44
1
$begingroup$
Yes, you need to create the package in a Linux environment.
$endgroup$
– Dan Carter
yesterday
1
$begingroup$
I'd guess any Linux OS would be okay, but not sure on that one. I ssh'd into an Amazon EC2 instance and created the package there...
$endgroup$
– Dan Carter
yesterday
1
$begingroup$
Yes, numpy, pandas, sklearn. It will work :) Just make sure you only add the packages you need, obviously with lambda you pay for memory used per second, package size may affect that.
$endgroup$
– Dan Carter
yesterday
|
show 3 more comments
$begingroup$
i tried this solution, however when I reference the zip on s3 it shows a dimension exceeded. Didi you face the same issue or am I doing anything wrong?
$endgroup$
– 3nomis
Mar 26 at 8:53
$begingroup$
How big is your deployment package uncompressed?
$endgroup$
– Dan Carter
Mar 26 at 9:44
1
$begingroup$
Yes, you need to create the package in a Linux environment.
$endgroup$
– Dan Carter
yesterday
1
$begingroup$
I'd guess any Linux OS would be okay, but not sure on that one. I ssh'd into an Amazon EC2 instance and created the package there...
$endgroup$
– Dan Carter
yesterday
1
$begingroup$
Yes, numpy, pandas, sklearn. It will work :) Just make sure you only add the packages you need, obviously with lambda you pay for memory used per second, package size may affect that.
$endgroup$
– Dan Carter
yesterday
$begingroup$
i tried this solution, however when I reference the zip on s3 it shows a dimension exceeded. Didi you face the same issue or am I doing anything wrong?
$endgroup$
– 3nomis
Mar 26 at 8:53
$begingroup$
i tried this solution, however when I reference the zip on s3 it shows a dimension exceeded. Didi you face the same issue or am I doing anything wrong?
$endgroup$
– 3nomis
Mar 26 at 8:53
$begingroup$
How big is your deployment package uncompressed?
$endgroup$
– Dan Carter
Mar 26 at 9:44
$begingroup$
How big is your deployment package uncompressed?
$endgroup$
– Dan Carter
Mar 26 at 9:44
1
1
$begingroup$
Yes, you need to create the package in a Linux environment.
$endgroup$
– Dan Carter
yesterday
$begingroup$
Yes, you need to create the package in a Linux environment.
$endgroup$
– Dan Carter
yesterday
1
1
$begingroup$
I'd guess any Linux OS would be okay, but not sure on that one. I ssh'd into an Amazon EC2 instance and created the package there...
$endgroup$
– Dan Carter
yesterday
$begingroup$
I'd guess any Linux OS would be okay, but not sure on that one. I ssh'd into an Amazon EC2 instance and created the package there...
$endgroup$
– Dan Carter
yesterday
1
1
$begingroup$
Yes, numpy, pandas, sklearn. It will work :) Just make sure you only add the packages you need, obviously with lambda you pay for memory used per second, package size may affect that.
$endgroup$
– Dan Carter
yesterday
$begingroup$
Yes, numpy, pandas, sklearn. It will work :) Just make sure you only add the packages you need, obviously with lambda you pay for memory used per second, package size may affect that.
$endgroup$
– Dan Carter
yesterday
|
show 3 more comments
Thanks for contributing an answer to Data Science Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
Use MathJax to format equations. MathJax reference.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f47955%2fsklearn-and-pandas-in-aws-lambda%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
