6

Keras model looks like this

inp = Input(shape=(maxlen, ))
x = Embedding(max_features, embed_size, weights=[embedding_matrix], trainable=False)(inp)
x = SpatialDropout1D(dropout)(x)
x = Bidirectional(LSTM(num_filters, return_sequences=True))(x)
max_pool = GlobalMaxPooling1D()(x)
x = concatenate([x_h, max_pool])
outp = Dense(6, activation="sigmoid")(x)

According to the documentation the output shape = input shape i.e. (samples, timesteps, channels)

Questions

  1. What does SpatialDropout1D() really do to the output of Embedding()? I know the output of LSTM Embedding is of dimension (batch_size, steps, features).
  2. Does SpatialDropout1D() just randomly replace some values of word embedding of each word by 0?
  3. How is SpatialDropout1D() different from Dropout() in Keras?
GeorgeOfTheRF
  • 2,078
  • 5
  • 18
  • 20

1 Answers1

1

Basically, it removes all the pixel in a row from all channels. eg: take [[1,1,1], [2,4,5]], there are 3 points with values in 2 channels, by doing SpatialDropout1D it zeros an entire row ie all attributes of a point is set to 0; like [[1,1,0], [2,4,0]]

number of such choices would be 3C0 + 3C1+ 3C2 + 3C3 = 8

The intuition behind this is in many cases for an image the adjacent pixels are correlated, so hiding one of them is not helping much, rather hiding entire row, that's gotta make a difference. reference material

Itachi
  • 251
  • 2
  • 8