Trjtdtk

How to count occurrences of text in a file?

How to avoid supervisors with prejudiced views?

What happened in Rome, when the western empire "fell"?

Do I need to write [sic] when a number is less than 10 but isn't written out?

Make solar eclipses exceedingly rare, but still have new moons

Why, when going from special to general relativity, do we just replace partial derivatives with covariant derivatives?

Why do remote US companies require working in the US?

Is it possible to replace duplicates of a character with one character using tr

Received an invoice from my ex-employer billing me for training; how to handle?

Can we say or write : "No, it'sn't"?

Domestic-to-international connection at Orlando (MCO)

Help understanding this unsettling image of Titan, Epimetheus, and Saturn's rings?

Flying from Cape Town to England and return to another province

Why don't programming languages automatically manage the synchronous/asynchronous problem?

What benefits would be gained by using human laborers instead of drones in deep sea mining?

Why is the US ranked as #45 in Press Freedom ratings, despite its extremely permissive free speech laws?

If a black hole is created from light, can this black hole then move at the speed of light?

I believe this to be a fraud - hired, then asked to cash check and send cash as Bitcoin

How to place nodes around a circle from some initial angle?

Would this house-rule that treats advantage as a +1 to the roll instead (and disadvantage as -1) and allows them to stack be balanced?

Won the lottery - how do I keep the money?

What connection does MS Office have to Netscape Navigator?

What does "Its cash flow is deeply negative" mean?

Can MTA send mail via a relay without being told so?

How to decide the processing power required based on the dataset?

1

$begingroup$

To train a machine learning model, the computer often needs more processing power. In this case, a powerful CPU is needed, since it is a large data set, it needs more memory, so rather than a CPU, GPU is the solution.

Do we need to decide which processor to use before we proceed? For example, will a 30000 sample data set need this much processing power?

Thanks in advance.

If any part of this question is not clear, please comment it.

machine-learning dataset

share|improve this question

edited Mar 25 at 8:13

Ethan

602224

asked Mar 24 at 5:21

PL_PathumPL_Pathum

1063

New contributor

PL_Pathum is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

$endgroup$

add a comment | 

1

$begingroup$

To train a machine learning model, the computer often needs more processing power. In this case, a powerful CPU is needed, since it is a large data set, it needs more memory, so rather than a CPU, GPU is the solution.

Do we need to decide which processor to use before we proceed? For example, will a 30000 sample data set need this much processing power?

Thanks in advance.

If any part of this question is not clear, please comment it.

machine-learning dataset

share|improve this question

edited Mar 25 at 8:13

Ethan

602224

asked Mar 24 at 5:21

PL_PathumPL_Pathum

1063

New contributor

PL_Pathum is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

$endgroup$

add a comment | 

$begingroup$

To train a machine learning model, the computer often needs more processing power. In this case, a powerful CPU is needed, since it is a large data set, it needs more memory, so rather than a CPU, GPU is the solution.

Do we need to decide which processor to use before we proceed? For example, will a 30000 sample data set need this much processing power?

Thanks in advance.

If any part of this question is not clear, please comment it.

machine-learning dataset

share|improve this question

edited Mar 25 at 8:13

Ethan

602224

asked Mar 24 at 5:21

PL_PathumPL_Pathum

1063

New contributor

PL_Pathum is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

$endgroup$

share|improve this question

edited Mar 25 at 8:13

Ethan

602224

asked Mar 24 at 5:21

PL_PathumPL_Pathum

1063

New contributor

PL_Pathum is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

share|improve this question

edited Mar 25 at 8:13

Ethan

602224

asked Mar 24 at 5:21

PL_PathumPL_Pathum

1063

New contributor

PL_Pathum is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

edited Mar 25 at 8:13

Ethan

602224

edited Mar 25 at 8:13

Ethan

602224

asked Mar 24 at 5:21

PL_PathumPL_Pathum

1063

New contributor

PL_Pathum is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

asked Mar 24 at 5:21

PL_PathumPL_Pathum

1063

1 Answer
1

active

oldest

votes

2

$begingroup$

Dataset (number of samples, number of features) is one variable. Algo/model complexity is another.

For example, linear regression will be much faster as compared to 4 layer neural network (and will require much lesser compute power).

So, before deciding need for compute powers, you can :

1. Try few models with hardware (or AWS instances) you already have


2. Estimate need for better hardware (CPU / GPU) based on the performance and what is the bottleneck

For very large data sets (say 10 TB+), I/O can become the bottleneck.

share|improve this answer

answered Mar 24 at 6:31

Shamit VermaShamit Verma

1,0991211

$endgroup$

• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  Got it, thank u. @ShamitVerma
  $endgroup$
  – PL_Pathum
  Mar 24 at 7:34
  

  
  
  
  

add a comment | 

Your Answer

StackExchange.ifUsing("editor", function ()
return StackExchange.using("mathjaxEditing", function ()
StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix)
StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["$", "$"], ["\\(","\\)"]]);
);
);
, "mathjax-editing");

StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "557"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);

else
createEditor();

);

function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);



);

PL_Pathum is a new contributor. Be nice, and check out our Code of Conduct.

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);

Sign up using Google

Sign up using Facebook

Sign up using Email and Password

Post as a guest

Name

Email

Required, but never shown

StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f47876%2fhow-to-decide-the-processing-power-required-based-on-the-dataset%23new-answer', 'question_page');

);

Post as a guest

Name

Email

Required, but never shown

2

$begingroup$

Dataset (number of samples, number of features) is one variable. Algo/model complexity is another.

For example, linear regression will be much faster as compared to 4 layer neural network (and will require much lesser compute power).

So, before deciding need for compute powers, you can :

1. Try few models with hardware (or AWS instances) you already have


2. Estimate need for better hardware (CPU / GPU) based on the performance and what is the bottleneck

For very large data sets (say 10 TB+), I/O can become the bottleneck.

share|improve this answer

answered Mar 24 at 6:31

Shamit VermaShamit Verma

1,0991211

$endgroup$

• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  Got it, thank u. @ShamitVerma
  $endgroup$
  – PL_Pathum
  Mar 24 at 7:34
  

  
  
  
  

add a comment | 

2

$begingroup$

Dataset (number of samples, number of features) is one variable. Algo/model complexity is another.

For example, linear regression will be much faster as compared to 4 layer neural network (and will require much lesser compute power).

So, before deciding need for compute powers, you can :

1. Try few models with hardware (or AWS instances) you already have


2. Estimate need for better hardware (CPU / GPU) based on the performance and what is the bottleneck

For very large data sets (say 10 TB+), I/O can become the bottleneck.

share|improve this answer

answered Mar 24 at 6:31

Shamit VermaShamit Verma

1,0991211

$endgroup$

• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  Got it, thank u. @ShamitVerma
  $endgroup$
  – PL_Pathum
  Mar 24 at 7:34
  

  
  
  
  

add a comment | 

$begingroup$

Dataset (number of samples, number of features) is one variable. Algo/model complexity is another.

For example, linear regression will be much faster as compared to 4 layer neural network (and will require much lesser compute power).

So, before deciding need for compute powers, you can :

1. Try few models with hardware (or AWS instances) you already have


2. Estimate need for better hardware (CPU / GPU) based on the performance and what is the bottleneck

For very large data sets (say 10 TB+), I/O can become the bottleneck.

share|improve this answer

answered Mar 24 at 6:31

Shamit VermaShamit Verma

1,0991211

$endgroup$

share|improve this answer

answered Mar 24 at 6:31

Shamit VermaShamit Verma

1,0991211

answered Mar 24 at 6:31

Shamit VermaShamit Verma

1,0991211

answered Mar 24 at 6:31

Shamit VermaShamit Verma

1,0991211