Trjtdtk

Can the Supreme Court Overturn an Impeachment?

When were female captains banned from Starfleet?

Travelling outside the UK without a passport

Where does the bonus feat in the cleric starting package come from?

Creepy dinosaur pc game identification

Why did the HMS Bounty go back to a time when whales are already rare?

Freedom of speech and where it applies

Why does the Sun have different day lengths, but not the gas giants?

Creature in Shazam mid-credits scene?

what is different between Do you interest vs interested in something?

Could the E-bike drivetrain wear down till needing replacement after 400 km?

Count the occurrence of each unique word in the file

Will the technology I first learn determine the direction of my future career?

Is it better practice to read straight from sheet music rather than memorize it?

Can I use Seifert-van Kampen theorem infinite times

Argument list too long when zipping large list of certain files in a folder

Can somebody explain the brexit thing in one or two child-proof sentences?

Why did the EU agree to delay the Brexit deadline?

How do you respond to a colleague from another team when they're wrongly expecting that you'll help them?

Is there an efficient solution to the travelling salesman problem with binary edge weights?

When a Cleric spontaneously casts a Cure Light Wounds spell, will a Pearl of Power recover the original spell or Cure Light Wounds?

Drawing ramified coverings with tikz

Non-trope happy ending?

Pre-mixing cryogenic fuels and using only one fuel tank

How feed a numpy array in batches in Keras

LSTM neural network for music generationMLP batch iteration in pythonComputing weights in batch gradient descentHow to improve my self-written Neural Network?How to work with large amount of data overcoming RAM issues in pythonFast introduction to deep learning in Python, with advanced math and some machine learning backgrounds, but not much Python experienceAre deep learning models way over the required capacity for their datasets' estimated entropies?Large Numpy.Array for Multi-label Image Classification (CelebA Dataset)Deep learning with Tensorflow: training with big data setsKeras Loss Value Extremely High + Prediction Result same

1

$begingroup$

I have the data in the following format:

1: DATA NUMPY ARRAY (trainX)

A numpy array of a set of numpy array of 3d np arrays.
To be more articulate the format is: [[3d data], [3d data], [3d data], [3d data], ...]

2: TARGET NUMPY ARRAY (trainY)

This consists of a numpy array of the corresponding target values for the above array.

The format is [target1, target2, target3]

The numpy array gets quite large, and considering that I'll be using a deep neural network, there will be many parameters that would need fitting into the memory as well.

How can I push the numpy arrays in batches for trainX and trainY

machine-learning neural-network deep-learning keras numpy

share|improve this question

edited Mar 19 at 15:07

Alireza Zolanvari

35716

asked Mar 19 at 14:53

thegravitythegravity

64

$endgroup$

add a comment | 

1

$begingroup$

I have the data in the following format:

1: DATA NUMPY ARRAY (trainX)

A numpy array of a set of numpy array of 3d np arrays.
To be more articulate the format is: [[3d data], [3d data], [3d data], [3d data], ...]

2: TARGET NUMPY ARRAY (trainY)

This consists of a numpy array of the corresponding target values for the above array.

The format is [target1, target2, target3]

The numpy array gets quite large, and considering that I'll be using a deep neural network, there will be many parameters that would need fitting into the memory as well.

How can I push the numpy arrays in batches for trainX and trainY

machine-learning neural-network deep-learning keras numpy

share|improve this question

edited Mar 19 at 15:07

Alireza Zolanvari

35716

asked Mar 19 at 14:53

thegravitythegravity

64

$endgroup$

add a comment | 

$begingroup$

I have the data in the following format:

1: DATA NUMPY ARRAY (trainX)

A numpy array of a set of numpy array of 3d np arrays.
To be more articulate the format is: [[3d data], [3d data], [3d data], [3d data], ...]

2: TARGET NUMPY ARRAY (trainY)

This consists of a numpy array of the corresponding target values for the above array.

The format is [target1, target2, target3]

The numpy array gets quite large, and considering that I'll be using a deep neural network, there will be many parameters that would need fitting into the memory as well.

How can I push the numpy arrays in batches for trainX and trainY

machine-learning neural-network deep-learning keras numpy

share|improve this question

edited Mar 19 at 15:07

Alireza Zolanvari

35716

asked Mar 19 at 14:53

thegravitythegravity

64

$endgroup$

share|improve this question

edited Mar 19 at 15:07

Alireza Zolanvari

35716

asked Mar 19 at 14:53

thegravitythegravity

64

share|improve this question

edited Mar 19 at 15:07

Alireza Zolanvari

35716

asked Mar 19 at 14:53

thegravitythegravity

64

edited Mar 19 at 15:07

Alireza Zolanvari

35716

edited Mar 19 at 15:07

Alireza Zolanvari

35716

asked Mar 19 at 14:53

thegravitythegravity

64

asked Mar 19 at 14:53

thegravitythegravity

64

1 Answer
1

active

oldest

votes

2

$begingroup$

You should implement a generator and feed it to model.fit_generator().

Your generator may look like this:

def batch_generator(X, Y, batch_size = BATCH_SIZE):
 indices = np.arange(len(X)) 
 batch=[]
 while True:
 # it might be a good idea to shuffle your data before each epoch
 np.random.shuffle(indices) 
 for i in indices:
 batch.append(i)
 if len(batch)==batch_size:
 yield X[batch], Y[batch]
 batch=[]

And then, somewhere in your code:

train_generator = batch_generator(trainX, trainY, batch_size = 64)
model.fit_generator(train_generator , ....)

UPD.:
I order to avoid placing all your data into memory beforehand, you can modify the generator to consume only the identifiers of your data-set and then load your data on-demand:

def batch_generator(ids, batch_size = BATCH_SIZE):
 batch=[]
 while True:
 np.random.shuffle(ids) 
 for i in ids:
 batch.append(i)
 if len(batch)==batch_size:
 yield load_data(batch)
 batch=[]

Your loader function may look like this:

def load_data(ids):
 X = []
 Y = []

 for i in ids:
 # read one or more samples from your storage, do pre-processing, etc.
 # for example:
 x = imread(f'image_i.jpg')
 ...
 y = targets[i]

 X.append(x)
 Y.append(y)

 return np.array(X), np.array(Y)

share|improve this answer

edited Mar 20 at 1:45

answered Mar 20 at 0:45

m0nzderrm0nzderr

663

$endgroup$

• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  Thanks so much for answering. Wouldn't batch_generator require X and Y as a numpy array already loaded in the memory which would still take up half the space. Is there a solution for that?
  $endgroup$
  – thegravity
  Mar 20 at 1:07
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  yes, there is a solution for that. please, see the updated comment.
  $endgroup$
  – m0nzderr
  Mar 20 at 1:36
  

  
  
  
  

add a comment | 

Your Answer

StackExchange.ifUsing("editor", function ()
return StackExchange.using("mathjaxEditing", function ()
StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix)
StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["$", "$"], ["\\(","\\)"]]);
);
);
, "mathjax-editing");

StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "557"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);

else
createEditor();

);

function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);



);

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);

Sign up using Google

Sign up using Facebook

Sign up using Email and Password

Post as a guest

Name

Email

Required, but never shown

StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f47623%2fhow-feed-a-numpy-array-in-batches-in-keras%23new-answer', 'question_page');

);

Post as a guest

Name

Email

Required, but never shown

2

$begingroup$

You should implement a generator and feed it to model.fit_generator().

Your generator may look like this:

def batch_generator(X, Y, batch_size = BATCH_SIZE):
 indices = np.arange(len(X)) 
 batch=[]
 while True:
 # it might be a good idea to shuffle your data before each epoch
 np.random.shuffle(indices) 
 for i in indices:
 batch.append(i)
 if len(batch)==batch_size:
 yield X[batch], Y[batch]
 batch=[]

And then, somewhere in your code:

train_generator = batch_generator(trainX, trainY, batch_size = 64)
model.fit_generator(train_generator , ....)

UPD.:
I order to avoid placing all your data into memory beforehand, you can modify the generator to consume only the identifiers of your data-set and then load your data on-demand:

def batch_generator(ids, batch_size = BATCH_SIZE):
 batch=[]
 while True:
 np.random.shuffle(ids) 
 for i in ids:
 batch.append(i)
 if len(batch)==batch_size:
 yield load_data(batch)
 batch=[]

Your loader function may look like this:

def load_data(ids):
 X = []
 Y = []

 for i in ids:
 # read one or more samples from your storage, do pre-processing, etc.
 # for example:
 x = imread(f'image_i.jpg')
 ...
 y = targets[i]

 X.append(x)
 Y.append(y)

 return np.array(X), np.array(Y)

share|improve this answer

edited Mar 20 at 1:45

answered Mar 20 at 0:45

m0nzderrm0nzderr

663

$endgroup$

• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  Thanks so much for answering. Wouldn't batch_generator require X and Y as a numpy array already loaded in the memory which would still take up half the space. Is there a solution for that?
  $endgroup$
  – thegravity
  Mar 20 at 1:07
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  yes, there is a solution for that. please, see the updated comment.
  $endgroup$
  – m0nzderr
  Mar 20 at 1:36
  

  
  
  
  

add a comment | 

2

$begingroup$

You should implement a generator and feed it to model.fit_generator().

Your generator may look like this:

def batch_generator(X, Y, batch_size = BATCH_SIZE):
 indices = np.arange(len(X)) 
 batch=[]
 while True:
 # it might be a good idea to shuffle your data before each epoch
 np.random.shuffle(indices) 
 for i in indices:
 batch.append(i)
 if len(batch)==batch_size:
 yield X[batch], Y[batch]
 batch=[]

And then, somewhere in your code:

train_generator = batch_generator(trainX, trainY, batch_size = 64)
model.fit_generator(train_generator , ....)

UPD.:
I order to avoid placing all your data into memory beforehand, you can modify the generator to consume only the identifiers of your data-set and then load your data on-demand:

def batch_generator(ids, batch_size = BATCH_SIZE):
 batch=[]
 while True:
 np.random.shuffle(ids) 
 for i in ids:
 batch.append(i)
 if len(batch)==batch_size:
 yield load_data(batch)
 batch=[]

Your loader function may look like this:

def load_data(ids):
 X = []
 Y = []

 for i in ids:
 # read one or more samples from your storage, do pre-processing, etc.
 # for example:
 x = imread(f'image_i.jpg')
 ...
 y = targets[i]

 X.append(x)
 Y.append(y)

 return np.array(X), np.array(Y)

share|improve this answer

edited Mar 20 at 1:45

answered Mar 20 at 0:45

m0nzderrm0nzderr

663

$endgroup$

• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  Thanks so much for answering. Wouldn't batch_generator require X and Y as a numpy array already loaded in the memory which would still take up half the space. Is there a solution for that?
  $endgroup$
  – thegravity
  Mar 20 at 1:07
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  yes, there is a solution for that. please, see the updated comment.
  $endgroup$
  – m0nzderr
  Mar 20 at 1:36
  

  
  
  
  

add a comment | 

$begingroup$

You should implement a generator and feed it to model.fit_generator().

Your generator may look like this:

def batch_generator(X, Y, batch_size = BATCH_SIZE):
 indices = np.arange(len(X)) 
 batch=[]
 while True:
 # it might be a good idea to shuffle your data before each epoch
 np.random.shuffle(indices) 
 for i in indices:
 batch.append(i)
 if len(batch)==batch_size:
 yield X[batch], Y[batch]
 batch=[]

And then, somewhere in your code:

train_generator = batch_generator(trainX, trainY, batch_size = 64)
model.fit_generator(train_generator , ....)

UPD.:
I order to avoid placing all your data into memory beforehand, you can modify the generator to consume only the identifiers of your data-set and then load your data on-demand:

def batch_generator(ids, batch_size = BATCH_SIZE):
 batch=[]
 while True:
 np.random.shuffle(ids) 
 for i in ids:
 batch.append(i)
 if len(batch)==batch_size:
 yield load_data(batch)
 batch=[]

Your loader function may look like this:

def load_data(ids):
 X = []
 Y = []

 for i in ids:
 # read one or more samples from your storage, do pre-processing, etc.
 # for example:
 x = imread(f'image_i.jpg')
 ...
 y = targets[i]

 X.append(x)
 Y.append(y)

 return np.array(X), np.array(Y)

share|improve this answer

edited Mar 20 at 1:45

answered Mar 20 at 0:45

m0nzderrm0nzderr

663

$endgroup$

def batch_generator(X, Y, batch_size = BATCH_SIZE):
 indices = np.arange(len(X)) 
 batch=[]
 while True:
 # it might be a good idea to shuffle your data before each epoch
 np.random.shuffle(indices) 
 for i in indices:
 batch.append(i)
 if len(batch)==batch_size:
 yield X[batch], Y[batch]
 batch=[]

train_generator = batch_generator(trainX, trainY, batch_size = 64)
model.fit_generator(train_generator , ....)

def batch_generator(ids, batch_size = BATCH_SIZE):
 batch=[]
 while True:
 np.random.shuffle(ids) 
 for i in ids:
 batch.append(i)
 if len(batch)==batch_size:
 yield load_data(batch)
 batch=[]

def load_data(ids):
 X = []
 Y = []

 for i in ids:
 # read one or more samples from your storage, do pre-processing, etc.
 # for example:
 x = imread(f'image_i.jpg')
 ...
 y = targets[i]

 X.append(x)
 Y.append(y)

 return np.array(X), np.array(Y)

share|improve this answer

edited Mar 20 at 1:45

answered Mar 20 at 0:45

m0nzderrm0nzderr

663

answered Mar 20 at 0:45

m0nzderrm0nzderr

663

answered Mar 20 at 0:45

m0nzderrm0nzderr

663