Derivative of Softmax loss function with $\text{one}_\text{hot}(y)$

Question

Given the following:

$h=g\left(w_1\cdot x+b_1\right)$

$ z=w_2\cdot h+b_2$

$y_\text{hat} = \mbox{softmax}\left(z\right)$

$\mbox{Loss}\left(y,y_\text{hat}\right)=-\sum \mbox{onehot}\left(y\right)\cdot \log\left(y_\text{hat}\right)$

How can we derivate the loss function: $\frac{\partial L}{\partial z}$?

What is $\operatorname{softmax}(z)$ and $onehot$? Make your OP clear by adding the meaning of your notation, otherwise your question will be doomed to be closed and forgotten. — Mittens, Jan 22 '22 at 20:46
What have you yourself attempted? What is the source of this question? — amWhy, Jan 28 '22 at 18:14

score 2 · Accepted Answer · answered Jan 23 '22 at 16:36

2

From here softmax, you will see that with your notations $$ \frac{\partial \mathrm{Loss}}{\partial \mathbf{z}} = \mathbf{y}_\mathrm{hat} - \mathrm{onehot}(y) $$ which is particularly simple...

answered Jan 23 '22 at 16:36

Steph

4,140
1
5
13

Derivative of Softmax loss function with $\text{one}_\text{hot}(y)$

1 Answers1