What is the proper way to implement a dropout layer in a CNN?

Question

Is this proceeding correct?

My intention was to add a dropout layer after concatenation, but to do so i needed to adjust the concat layer's output to the appropriate shape (samples, timesteps, channels), hence expanding the dimension from (None, 4096) to (None, 1, 4096) and consequently undo the operation after the output.

score 3 · Answer 1 · answered Jan 19 '22 at 16:30

Instead of explicitly adding a new dimension, in tensorflow 2.7.0 you could just use keepdims=True as an argument of the GlobalAveragePooling2D layer.

Example:

def TestModel():
  # specify the input shape
  in_1 = tf.keras.layers.Input(shape = (256,256,3))
  in_2 = tf.keras.layers.Input(shape = (256,256,3))

  x1 = tf.keras.layers.Conv2D(64, (3,3))(in_1)
  x1 = tf.keras.layers.LeakyReLU()(x1)
  x1 = tf.keras.layers.GlobalAveragePooling2D(keepdims = True)(x1)

  x2 = tf.keras.layers.Conv2D(64, (3,3))(in_2)
  x2 = tf.keras.layers.LeakyReLU()(x2)
  x2 = tf.keras.layers.GlobalAveragePooling2D(keepdims = True)(x2)
 
  x = tf.keras.layers.concatenate([x1,x2])
  x = tf.keras.layers.SpatialDropout2D(0.2)(x)
  x = tf.keras.layers.Dense(1000)(x)

  # create the model
  model = tf.keras.Model(inputs=(in_1,in_2), outputs=x)

  return model

#Testcode
model = TestModel()
model.summary()
tf.keras.utils.plot_model(model, show_shapes=True, expand_nested=False, show_dtype=True, to_file="model.png")

If you require to squeeze it in the end, you could still do it.

Thank you very much. The problem is that the output of my avg pool layer has shape (None, 2048), this way i don't see how preserving dims could help me cause i would have to expand it. Anyway thank you for the tip, i didn't knew this one. — Luciano Dourado, Jan 21 '22 at 00:11

AloneTogether · Accepted Answer · 2022-01-19T17:20:36.747

1

If you plan to use the SpatialDropout1D layer, it has to receive a 3D tensor (batch_size, time_steps, features), so adding an additional dimension to your tensor before feeding it to the dropout layer is one option that is perfectly legitimate. Note, though, that in your case you could use both SpatialDropout1D or Dropout:

import tensorflow as tf

samples = 2
timesteps = 1
features = 5
x = tf.random.normal((samples, timesteps, features))
s = tf.keras.layers.SpatialDropout1D(0.5)
d = tf.keras.layers.Dropout(0.5)

print(s(x, training=True))
print(d(x, training=True))

tf.Tensor(
[[[-0.5976591  1.481788   0.         0.         0.       ]]

 [[ 0.        -4.6607018 -0.         0.7036132  0.       ]]], shape=(2, 1, 5), dtype=float32)
tf.Tensor(
[[[-0.5976591  1.481788   0.5662646  2.8400114  0.9111476]]

 [[ 0.        -0.        -0.         0.7036132  0.       ]]], shape=(2, 1, 5), dtype=float32)

I think that SpatialDropout1D layers are most suitable after CNN layers.

edited Jan 19 '22 at 17:20

answered Jan 19 '22 at 16:14

AloneTogether

25,814
5
20
39

Thank you very much, i will do a little bit more research about the difference between them. The thing here is that i have a concatenation between a feature extraction model and a model that is being trained by considering these additional features from a pre-trained model (both resnets). In this case i don't know yet which one should be better for this application. – Luciano Dourado Jan 21 '22 at 00:13
Besides that i have searched but without much success. Do you know whay does the arg **time_steps** means? – Luciano Dourado Jan 21 '22 at 00:14
1

It is just a spatial / temporal dimension. Check this post for example https://stackoverflow.com/questions/69591717/how-is-the-keras-conv1d-input-specified-i-seem-to-be-lacking-a-dimension/69594400#69594400. – AloneTogether Jan 21 '22 at 06:45

What is the proper way to implement a dropout layer in a CNN?

2 Answers2