1

The problem is defined as $$ \min_{w} = \sum_{i=1}^{n} \sum_{j=1}^{n}\left\{ \left(\sum_{k=1}^{3}w_kP_{ij}^{(k)}\right) \log \left(\sum_{k=1}^{3}w_kP_{ij}^{(k)}\right) \right\} + \gamma \|w\|_2^2\\ $$ and the constraints are $$ w_k\gt 0,~~\sum_{k=1}^{3} w_k = 1 $$ where $w=[w_1, w_2, w_3]^T \in R^{3\times 1}$ and $0\le P_{ij}^{(k)} \le 1$.

My basic idea on this problem is as follows:

  1. the gradient on $w_k$ is $$ \nabla_{w_{k}} = \sum_{i=1}^{n} \sum_{j=1}^{n} \left\{P_{ij}^{(k)} \log \left(\sum_{k'=1}^{3}w_{k'} P_{ij}^{(k') } \right) + P_{ij}^{(k)} \right\}+2\gamma w_k $$

then, I don't know how to do next. How could I solve this problem with the gradient projection method? Or could you help provide some references on how to solve this problem? Thanks!

Royi
  • 10,050
mining
  • 231
  • 1
    You need a projected gradient algorithm at the very least to handle the simplex constraint. Straight gradient will fail. – Michael Grant Apr 11 '15 at 12:58
  • Hi, Prof. @MichaelGrant, thank you! I'm trying to solve this problem with the Lagrangian method. – mining Apr 11 '15 at 14:19

1 Answers1

1

The problem can be formulated as:

$$ \arg \min_{\boldsymbol{w}} \sum_{l = 1}^{{n}^{2}} \boldsymbol{p}_{l}^{T} \boldsymbol{w} \log \left( \boldsymbol{p}_{l}^{T} \boldsymbol{w} \right) + \frac{\gamma}{2} {\left\| \boldsymbol{w} \right\|}_{2}^{2}, \; \text{ subject to } \boldsymbol{w} \in \mathcal{\Delta}^{n} $$

Where $\mathcal{\Delta}^{n}$ is the Unit Simplex Ball.

In order to use the Projected Gradient Descent one needs 2 components:

  • The Gradient: ${\nabla}_{\boldsymbol{w}} f \left( \boldsymbol{w} \right) = \sum_{l = 1}^{{n}^{2}} \boldsymbol{p}_{l} + \log \left( \boldsymbol{p}_{l}^{T} \boldsymbol{w} \right) \boldsymbol{p}_{l} + \gamma \boldsymbol{w}$.
  • The Pojection: The projection onto the Unit Simplex has no closed form. Yet it has some efficient methods to calculate. See Orthogonal Projection onto the Unit Simplex.
Royi
  • 10,050