Keras Multiple “Softmax” in last layer possible?

Question

Is it possible to implement mutiple softmaxes in the last layer in Keras? So the sum of Nodes 1-4 = 1; 5-8 = 1; etc.

Should I go for a different network design?

Answer 1

I would use the functional interface.

Something like this:

from keras.layers import Activation, Input, Dense
from keras.models import Model
from keras.layers.merge import Concatenate

input_ = Input(shape=input_shape)

x = input_
x1 = Dense(4, x)
x2 = Dense(4, x)
x3 = Dense(4, x)
x1 = Activation('softmax')(x1)
x2 = Activation('softmax')(x2)
x3 = Activation('softmax')(x3)
x = Concatenate([x1, x2, x3])

model = Model(inputs=input_, outputs=x)

Answer 2

It is possible just implement your own softmax function. You can split a tensor to parts, then compute softmax separately per part and concatenate tensor parts:

def custom_softmax(t):
    sh = K.shape(t)
    partial_sm = []
    for i in range(sh[1] // 4):
        partial_sm.append(K.softmax(t[:, i*4:(i+1)*4]))
    return K.concatenate(partial_sm)

concatenate without axis argument concatenate through last axis (in our case axis=1).

Then you can include this activation function in a hidden layer or add it to a graph.

Dense(activation=custom_activation)

or

model.add(Activation(custom_activation))

You also need to define a new cost function.