Trjtdtk

I am a newbie in GPU based training and Deep learning models. I am running cDCGAN (Conditonal DCGAN) in tensorflow on my 2 Nvidia GTX 1080 GPU's. My data-set consists of around 32,0000 images with size 64*64 and 2350 class labels. If I set my batch size 32 or large I get OOM error like below. So I am using 10 batch size for now.

tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[32,64,64,2351] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
 [[Node: discriminator/concat = ConcatV2[N=2, T=DT_FLOAT, Tidx=DT_INT32, _device="/job:localhost/replica:0/task:0/device:GPU:0"](_arg_Placeholder_0_0/_41, _arg_Placeholder_3_0_3/_43, discriminator/concat/axis)]]
Caused by op 'discriminator/concat', defined at:
 File "cdcgan.py", line 221, in <module>
 D_real, D_real_logits = discriminator(x, y_fill, isTrain)
 File "cdcgan.py", line 48, in discriminator
 cat1 = tf.concat([x, y_fill], 3)

The training is very slow which I understand is down to the batch size (correct me if I am wrong). If I do help -n 1 nvidia-smi, I get the following output.

The GPU:0 is mainly used, as the Volatile GPU-Util gives me around 0%-65% whereas GPU:1 is always 0%-3% max. Performance for GPU:0 is always in P2 whereas GPU:1 is mostly P8 or sometimes P2. I have the following questions.

1) Why I am getting OOM error on the large batch size although my dataset and model are not that big?

2) How can I utilize both GPU's equally in Tensorflow so that the performance is fast? (From the above error, it looks like GPU:0 gets full immediately whereas GPU:1 is not fully utilized. it's my understanding only)

Model Details are as follows

Generator:

I have 4 layers (fully connected, UpSampling2d-conv2d, UpSampling2d-conv2d, conv2d).

W1 is of the shape [X+Y, 16*16*128] i.e. (2450, 32768), w2 [3, 3, 128, 64], w3 [3, 3, 64, 32], w4 [[3, 3, 32, 1]] respectively

Discriminator

It has five layers (conv2d, conv2d, conv2d, conv2d, fully connected).

w1 [5, 5, X+Y, 64] i.e. (5, 5, 2351, 64), w2 [3, 3, 64, 64], w3 [3, 3, 64, 128], w4 [2, 2, 128, 256], [16*16*256, 1] respectively.

Session Configuration
I am also allocating memory in advance via

gpu_options = tf.GPUOptions(allow_growth=True)
session = tf.InteractiveSession(config=tf.ConfigProto(gpu_options=gpu_options))

1) Why I am getting OOM error on the large batch size although my dataset and model are not that big?

Yes, the batch size is probably the reason.

Also, another the reason is that you don't use the second GPU at all (otherwise, both GPUs will split the batch - computation - and you could use larger batches).

2) How can I utilize both GPU's equally in Tensorflow so that the performance is fast? (From the above error, it looks like GPU:0 gets full immediately whereas GPU:1 is not fully utilized. it's my understanding only)

By default, Tensorflow occupies all available GPUs (that's way you see it with nvidia-smi - you have a process 34589 which took both GPUs), but, unless you specify in the code to actually use multi GPUs, it will use only one by default.

Here is an official TF docs about how to use multi GPUs: https://www.tensorflow.org/guide/using_gpu#using_multiple_gpus

Here is some tutorial for multi-gpu usage with more examples: https://jhui.github.io/2017/03/07/TensorFlow-GPU/