Trjtdtk

What is the motivation for a law requiring 2 parties to consent for recording a conversation

How can I make payments on the Internet without leaving a money trail?

How to Override Magento 2 vendor files

On the insanity of kings as an argument against monarchy

Are USB sockets on wall outlets live all the time, even when the switch is off?

Differentiate between line ending within polygon and line passing all the way through polygon - QGIS

If the Wish spell is used to duplicate the effect of Simulacrum, are existing duplicates destroyed?

"as much details as you can remember"

Process LaTeX code only if package is loaded

Geography at the pixel level

Can I connect a DC high voltage booster directly to my Raspberry Pi?

Dual Citizen. Exited the US on Italian passport recently

Falsification in Math vs Science

How to manage monthly salary

What is the purpose of the constant in the probability density function

Which Sci-Fi work first showed weapon of galactic-scale mass destruction?

Can the Protection from Evil and Good spell be used on the caster?

Must numeric JSON keys be quoted?

What tool would a Roman-age civilization have to grind silver and other metals into dust?

Not able to paste multiple components

How can I fix this gap between bookcases I made?

Is three citations per paragraph excessive for undergraduate research paper?

Flying Bloodthirsty Lampshades

What do hard-Brexiteers want with respect to the Irish border?

Keras Tensorboard callback stops working after a couple thousand batches

0

$begingroup$

I'm training a model using Keras, and I've written a custom callback, F1Metrics that computes the average F1 score every 1200 batches. In the callback, I write the score to a file, and also append it to the logs of the model, to be later displayed with Tensorboard.

F1Metrics.on_batch_end:

def on_batch_end(self, batch, logs=):
 if not self.batch_save:
 return

 self.batch += 1

 if self.batch == 0:
 return

 if self.batch % self.N != 0:
 return

 df, val_loss = self.calculate_metrics_table(
 self.model,
 self.labels,
 self.number_of_classes,
 self.generator,
 'ep0_b1'.format(self.epoch, batch),
 val_loss_fct=self.val_loss_fct,
 loss=logs['loss'],
 class_names=self.class_names,
 )

 df.to_csv(
 os.path.join(
 self.output_path,
 'f1_metrics_epoch_0_batch_1_var_thresh.csv'.format(
 self.epoch, batch)),
 index=False)

 logs['AUC'] = df['AUC'].mean()
 logs['valid_val_loss'] = val_loss

Tensorboard callback:

tensorboard = TensorBoard(
 log_dir=str(model_logs_output_dir),
 batch_size = hyperparameters['batch_size'],
 histogram_freq=0,
 update_freq=hyperparameters['batch_size']*hyperparameters['batch_interval'],
)

The files are created for the entire duration of the training. But after ~25k batches, the updated value stops being recorded and displayed by Tensorboard, as seen below. The batch loss I assume is being calculated by Keras.

Do you have any idea why this happens?

keras tensorflow

share|improve this question

asked Mar 29 at 11:56

Tenescu AndreiTenescu Andrei

113

$endgroup$

add a comment | 

0

$begingroup$

I'm training a model using Keras, and I've written a custom callback, F1Metrics that computes the average F1 score every 1200 batches. In the callback, I write the score to a file, and also append it to the logs of the model, to be later displayed with Tensorboard.

F1Metrics.on_batch_end:

def on_batch_end(self, batch, logs=):
 if not self.batch_save:
 return

 self.batch += 1

 if self.batch == 0:
 return

 if self.batch % self.N != 0:
 return

 df, val_loss = self.calculate_metrics_table(
 self.model,
 self.labels,
 self.number_of_classes,
 self.generator,
 'ep0_b1'.format(self.epoch, batch),
 val_loss_fct=self.val_loss_fct,
 loss=logs['loss'],
 class_names=self.class_names,
 )

 df.to_csv(
 os.path.join(
 self.output_path,
 'f1_metrics_epoch_0_batch_1_var_thresh.csv'.format(
 self.epoch, batch)),
 index=False)

 logs['AUC'] = df['AUC'].mean()
 logs['valid_val_loss'] = val_loss

Tensorboard callback:

tensorboard = TensorBoard(
 log_dir=str(model_logs_output_dir),
 batch_size = hyperparameters['batch_size'],
 histogram_freq=0,
 update_freq=hyperparameters['batch_size']*hyperparameters['batch_interval'],
)

The files are created for the entire duration of the training. But after ~25k batches, the updated value stops being recorded and displayed by Tensorboard, as seen below. The batch loss I assume is being calculated by Keras.

Do you have any idea why this happens?

keras tensorflow

share|improve this question

asked Mar 29 at 11:56

Tenescu AndreiTenescu Andrei

113

$endgroup$

add a comment | 

$begingroup$

I'm training a model using Keras, and I've written a custom callback, F1Metrics that computes the average F1 score every 1200 batches. In the callback, I write the score to a file, and also append it to the logs of the model, to be later displayed with Tensorboard.

F1Metrics.on_batch_end:

def on_batch_end(self, batch, logs=):
 if not self.batch_save:
 return

 self.batch += 1

 if self.batch == 0:
 return

 if self.batch % self.N != 0:
 return

 df, val_loss = self.calculate_metrics_table(
 self.model,
 self.labels,
 self.number_of_classes,
 self.generator,
 'ep0_b1'.format(self.epoch, batch),
 val_loss_fct=self.val_loss_fct,
 loss=logs['loss'],
 class_names=self.class_names,
 )

 df.to_csv(
 os.path.join(
 self.output_path,
 'f1_metrics_epoch_0_batch_1_var_thresh.csv'.format(
 self.epoch, batch)),
 index=False)

 logs['AUC'] = df['AUC'].mean()
 logs['valid_val_loss'] = val_loss

Tensorboard callback:

tensorboard = TensorBoard(
 log_dir=str(model_logs_output_dir),
 batch_size = hyperparameters['batch_size'],
 histogram_freq=0,
 update_freq=hyperparameters['batch_size']*hyperparameters['batch_interval'],
)

The files are created for the entire duration of the training. But after ~25k batches, the updated value stops being recorded and displayed by Tensorboard, as seen below. The batch loss I assume is being calculated by Keras.

Do you have any idea why this happens?

keras tensorflow

share|improve this question

asked Mar 29 at 11:56

Tenescu AndreiTenescu Andrei

113

$endgroup$

def on_batch_end(self, batch, logs=):
 if not self.batch_save:
 return

 self.batch += 1

 if self.batch == 0:
 return

 if self.batch % self.N != 0:
 return

 df, val_loss = self.calculate_metrics_table(
 self.model,
 self.labels,
 self.number_of_classes,
 self.generator,
 'ep0_b1'.format(self.epoch, batch),
 val_loss_fct=self.val_loss_fct,
 loss=logs['loss'],
 class_names=self.class_names,
 )

 df.to_csv(
 os.path.join(
 self.output_path,
 'f1_metrics_epoch_0_batch_1_var_thresh.csv'.format(
 self.epoch, batch)),
 index=False)

 logs['AUC'] = df['AUC'].mean()
 logs['valid_val_loss'] = val_loss

tensorboard = TensorBoard(
 log_dir=str(model_logs_output_dir),
 batch_size = hyperparameters['batch_size'],
 histogram_freq=0,
 update_freq=hyperparameters['batch_size']*hyperparameters['batch_interval'],
)

share|improve this question

asked Mar 29 at 11:56

Tenescu AndreiTenescu Andrei

113

share|improve this question

asked Mar 29 at 11:56

Tenescu AndreiTenescu Andrei

113

asked Mar 29 at 11:56

Tenescu AndreiTenescu Andrei

113

asked Mar 29 at 11:56

Tenescu AndreiTenescu Andrei

113