Trjtdtk

So when running this example script from Keras repo (https://github.com/keras-team/keras/blob/master/examples/lstm_seq2seq.py), I found that we can easily run into out of memory for the input or output one-hot encoding in this code:

encoder_input_data = np.zeros(
 (len(input_texts), max_encoder_seq_length, num_encoder_tokens),
 dtype='float32')
decoder_input_data = np.zeros(
 (len(input_texts), max_decoder_seq_length, num_decoder_tokens),
 dtype='float32')
decoder_target_data = np.zeros(
 (len(input_texts), max_decoder_seq_length, num_decoder_tokens),
dtype='float32')

When I have a huge size of training samples, this one-hot does not fit into memory. Is there way to handle this issue?

UPDATE:
I changed float32 to uint, but thats about the smallest one hot array can get.

so I found a solution by dividing input training data into small batches and it did the trick. Here is the code:

divideby=1000
for j in range(divideby):
 start=j*len(input_texts)/divideby
 end=(j+1)*len(input_texts)/divideby if j<divideby+1 else -1

 encoder_input_data = np.zeros(
 (end-start, max_encoder_seq_length, num_encoder_tokens),
 dtype='uint8') # size_of_training_samples, max_length_word_measured_in_characters,number_of_unique_chars,
 decoder_input_data = np.zeros(
 (end-start, max_decoder_seq_length, num_decoder_tokens),
 dtype='uint8')
 decoder_target_data = np.zeros(
 (end-start, max_decoder_seq_length, num_decoder_tokens),
 dtype='uint8')

 for i, (input_text, target_text) in enumerate(zip(input_texts[start:end], target_texts[start:end])):
 for t, char in enumerate(input_text.split(splitby)):
 encoder_input_data[i, t, input_token_index[char]] = 1.
 for t, char in enumerate(target_text.split(splitby)):
 # decoder_target_data is ahead of decoder_input_data by one timestep
 decoder_input_data[i, t, target_token_index[char]] = 1.
 if t > 0:
 # decoder_target_data will be ahead by one timestep
 # and will not include the start character.
 decoder_target_data[i, t - 1, target_token_index[char]] = 1.

 model.fit([encoder_input_data, decoder_input_data], decoder_target_data,
 batch_size=batch_size,
 epochs=epochs,
 validation_split=0.2)
```