Trjtdtk

Significance of Cersei's obsession with elephants?

Does the Weapon Master feat grant you a fighting style?

Chinese Seal on silk painting - what does it mean?

What font is "z" in "z-score"?

Why aren't air breathing engines used as small first stages?

When a candle burns, why does the top of wick glow if bottom of flame is hottest?

Dating a Former Employee

Is it a good idea to use CNN to classify 1D signal?

Sum letters are not two different

Do wooden building fires get hotter than 600°C?

SF book about people trapped in a series of worlds they imagine

If my PI received research grants from a company to be able to pay my postdoc salary, did I have a potential conflict interest too?

How do living politicians protect their readily obtainable signatures from misuse?

Is this homebrew Lady of Pain warlock patron balanced?

What does the "x" in "x86" represent?

Has negative voting ever been officially implemented in elections, or seriously proposed, or even studied?

Can anything be seen from the center of the Boötes void? How dark would it be?

How would a mousetrap for use in space work?

Did MS DOS itself ever use blinking text?

How does the math work when buying airline miles?

How do I use the new nonlinear finite element in Mathematica 12 for this equation?

How to tell that you are a giant?

What is the effect of altitude on true airspeed?

What is the meaning of the simile “quick as silk”?

Training model in Keras(TF backend) with GPU

1

$begingroup$

I want to train my custom dataset for licence plate detection. I have 2000 image for this. First, I trained my labelling dataset in Keras (TF backend) with CPU.

optimizer : Adam, learning rate : 0.01 , batch size = 64,
iteration:100000

In this case, my training period exceeded 10 days, loss is 10. When I tested the model could detect licence plates. (acc : 80%)

Then, I decided to train with gpu because of the training period is too long.

I uninstall tensorflow and I install tensorflow-gpu(1.13), Cuda(10.0) and Cudnn(7.4) and I added these codes:

config = tf.ConfigProto()
config.gpu_options.allow_growth = True
session = tf.InteractiveSession(config=config)

In this time, when I use the same values (Adam, lr=0.01, itr=100000, bs=64), loss value decreases very fast every iteration (loss begins with 210 and ends 0.000000). The training ends in four-five hours. But when I tested, my model could not detect any licence plate.

I try to change values. I did learning rate = 0.000001, iteration =3000 so that the loss is 0.7, training ends 10-20 minutes, but model detects very wrong. (and I know this number is very bad for learning rate)

My GPU : GeForce RTX 2080

How can I optimize these numbers, where is my mistake?

Thank you.

keras tensorflow gpu

share|improve this question

asked Apr 3 at 9:11

little_learning_ratelittle_learning_rate

82

$endgroup$

• 
  
  
  

  
  3
  

  
  
  
  
  
  

  
  $begingroup$
  The fact you perform the training on a CPU or on a GPU wouldn't affect the results, more over from 80% to 0%. It seems to me, than when you are using the GPU, you are not testing the model with the learned weights... Are you sure you are not closing the tf.Session() or you are not restoring the weights?
  $endgroup$
  – ignatius
  Apr 3 at 9:24
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  thank you, I think so too, the training on a CPU or on a GPU souldn't affect the results. but according to my results it does. actually I dont understand why is the final loss value is different when training with Gpu and Cpu with the same values? i think my learning rate and batch size values are wrong, because when I changed these numbers training with gpu, my model can detect something but wrong, at least it tries. (so, the problem should not relate to weights) i dont know how to optimize learning rate and batch size training with gpu.
  $endgroup$
  – little_learning_rate
  Apr 3 at 11:06
  

  
  
  
  

add a comment | 

1

$begingroup$

I want to train my custom dataset for licence plate detection. I have 2000 image for this. First, I trained my labelling dataset in Keras (TF backend) with CPU.

optimizer : Adam, learning rate : 0.01 , batch size = 64,
iteration:100000

In this case, my training period exceeded 10 days, loss is 10. When I tested the model could detect licence plates. (acc : 80%)

Then, I decided to train with gpu because of the training period is too long.

I uninstall tensorflow and I install tensorflow-gpu(1.13), Cuda(10.0) and Cudnn(7.4) and I added these codes:

config = tf.ConfigProto()
config.gpu_options.allow_growth = True
session = tf.InteractiveSession(config=config)

In this time, when I use the same values (Adam, lr=0.01, itr=100000, bs=64), loss value decreases very fast every iteration (loss begins with 210 and ends 0.000000). The training ends in four-five hours. But when I tested, my model could not detect any licence plate.

I try to change values. I did learning rate = 0.000001, iteration =3000 so that the loss is 0.7, training ends 10-20 minutes, but model detects very wrong. (and I know this number is very bad for learning rate)

My GPU : GeForce RTX 2080

How can I optimize these numbers, where is my mistake?

Thank you.

keras tensorflow gpu

share|improve this question

asked Apr 3 at 9:11

little_learning_ratelittle_learning_rate

82

$endgroup$

• 
  
  
  

  
  3
  

  
  
  
  
  
  

  
  $begingroup$
  The fact you perform the training on a CPU or on a GPU wouldn't affect the results, more over from 80% to 0%. It seems to me, than when you are using the GPU, you are not testing the model with the learned weights... Are you sure you are not closing the tf.Session() or you are not restoring the weights?
  $endgroup$
  – ignatius
  Apr 3 at 9:24
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  thank you, I think so too, the training on a CPU or on a GPU souldn't affect the results. but according to my results it does. actually I dont understand why is the final loss value is different when training with Gpu and Cpu with the same values? i think my learning rate and batch size values are wrong, because when I changed these numbers training with gpu, my model can detect something but wrong, at least it tries. (so, the problem should not relate to weights) i dont know how to optimize learning rate and batch size training with gpu.
  $endgroup$
  – little_learning_rate
  Apr 3 at 11:06
  

  
  
  
  

add a comment | 

$begingroup$

I want to train my custom dataset for licence plate detection. I have 2000 image for this. First, I trained my labelling dataset in Keras (TF backend) with CPU.

optimizer : Adam, learning rate : 0.01 , batch size = 64,
iteration:100000

In this case, my training period exceeded 10 days, loss is 10. When I tested the model could detect licence plates. (acc : 80%)

Then, I decided to train with gpu because of the training period is too long.

I uninstall tensorflow and I install tensorflow-gpu(1.13), Cuda(10.0) and Cudnn(7.4) and I added these codes:

config = tf.ConfigProto()
config.gpu_options.allow_growth = True
session = tf.InteractiveSession(config=config)

In this time, when I use the same values (Adam, lr=0.01, itr=100000, bs=64), loss value decreases very fast every iteration (loss begins with 210 and ends 0.000000). The training ends in four-five hours. But when I tested, my model could not detect any licence plate.

I try to change values. I did learning rate = 0.000001, iteration =3000 so that the loss is 0.7, training ends 10-20 minutes, but model detects very wrong. (and I know this number is very bad for learning rate)

My GPU : GeForce RTX 2080

How can I optimize these numbers, where is my mistake?

Thank you.

keras tensorflow gpu

share|improve this question

asked Apr 3 at 9:11

little_learning_ratelittle_learning_rate

82

$endgroup$

config = tf.ConfigProto()
config.gpu_options.allow_growth = True
session = tf.InteractiveSession(config=config)

share|improve this question

asked Apr 3 at 9:11

little_learning_ratelittle_learning_rate

82

share|improve this question

asked Apr 3 at 9:11

little_learning_ratelittle_learning_rate

82

asked Apr 3 at 9:11

little_learning_ratelittle_learning_rate

82

asked Apr 3 at 9:11

little_learning_ratelittle_learning_rate

82