1

Given an $n\times n$ real diagonal matrix $D$ and an $m\times m$ real diagonal matrix $W$ (where $n\geq m$) with $\text{tr}(W^2)=1$, consider the following optimization problem in $X \in \mathbb{R}^{n \times m}$

$$\begin{array}{ll} \text{minimize} & \text{tr}\!\left((XW)^TD\,(XW)\right)\\ \text{subject to} & X^T X = I_m\end{array}$$

In the "equal-weighted" case $W=(I_m/m)^{1/2}$, this reduces to a standard Rayleigh quotient minimization problem. But in the more general case, setting up and solving the Lagrangian is causing some difficulties.

My current approach: Rewrite the constraint as $W^TX^TXW = W^2$, then set up the Lagrangian

$$\mathcal{L}(X,\Lambda) = \text{tr}\!\left((XW)^TD\,(XW)\right) + \text{tr}\!\left(\Lambda(W^2-W^TX^TXW)\right)$$

The first term enters into the first-order condition as $2DXW^2$, but I'm having trouble differentiating the second term since this formulation doesn't appear in any of the "cookbooks."

I have a hunch that given the orthonormality constraint $X^TX = I_m$, the solution doesn't depend on the matrix $W$ (so that one solution is given by $X$ equal to the first $m$ columns of $I_n$, since these can be taken as the eigenvectors for the diagonal matrix $D$), but am not positive. Any help appreciated.

Rodrigo de Azevedo
  • 23,223
sunga
  • 11
  • 3
  • gradient of the second term is: $-XW(\Lambda + \Lambda^T)W^T$ – user550103 Jul 31 '19 at 05:08
  • @user550103 so just working through the algebra, we get $2DXW^2 = 2XW\Lambda W$ (since we can take $\Lambda$ to be symmetric), equivalently $XW = DXW\Lambda^{-1}$. Plugging into the constraint, $WX^TDXW\Lambda^{-1} = W^2$, or $\Lambda = W^{-1}X^TDXW$. Plugging that back into the first-order condition, $DXW = XWW^{-1}X^TDXW = XX^TDXW$, for which one solution is $X = \begin{bmatrix} I_m \ 0_{n-m,m} \end{bmatrix}$, as conjectured. Anything off about that logic? – sunga Jul 31 '19 at 07:35

1 Answers1

0

The usual arguments apply. By padding $X$ and $W$ with zeros, you may assume that $X,D,W$ are square matrices of the same sizes. Then $$ \operatorname{tr}((XW)^TD(XW))=\operatorname{tr}(X^TDXW^2)=\sum_{i,j}d_iw_j^2x_{ij}^2 $$ is linear in the entries of the orthostochastic matrix $X\circ X$. By Birkhoff's theorem, $X\circ X$ is a convex combination of permutation matrices. Therefore the matrix trace in question is minimised when $X$ is a permutation matrix such that $x_{i\sigma(i)}=1$ for some permutation $\sigma\in S_n$. The minimisation problem thus boils down to $$ \min_{\sigma\in S_n} \sum_i d_iw_{\sigma(i)}^2. $$ Without loss of generality, we may assume that $d_1\ge d_2\ge\cdots\ge d_n$. Then a global minimiser is obviously given by any $\sigma\in S_n$ that makes $w_{\sigma(1)}^2\le w_{\sigma(2)}^2\le\cdots\le w_{\sigma(n)}^2$. Now, by extracting the first $m$ columns of $X$, one obtains a minimiser for the original minimisation problem.

user1551
  • 149,263
  • This makes sense, but it suggests that my comment on my original post is correct if and only if $w_1^2\leq w_2^2\leq\cdots\leq w_m^2$, since this is the only case in which the global minimizer features $x_{ii}=1,x_{i,j\neq i}=0$, as in my comment. But my comment proceeds fairly straightforwardly according to the "less elegant approach" in your link, so I don't see what's incorrect about the steps taken there. Any thoughts? – sunga Jul 31 '19 at 21:16
  • For the purposes of my research question, this is actually a subproblem of a larger problem, in which we get to choose both the values ${w_j}$ and the values ${x_{ij}}$ (s.t. all the constraints listed). Then given the ordering $d_1\geq d_2\cdots \geq d_n$, if we have an ordering of the values ${w_j}$ that's not weakly increasing, we can in fact do better with respect to our subproblem minimization objective by rearranging them to be weakly increasing (while keeping the set of values the same). This implies that the global minimizer does in fact have to feature $x_{ii}=1$, I think? – sunga Jul 31 '19 at 22:19
  • @sunga To be frank, I don't quite follow what you are doing. You seem to assert that $X=\pmatrix{I\ 0}$ gives you a local minimum, but the Lagrange multipliers method can give you a local maximum as well. Without any justification, I don't see why your $X$ has to be a local (or even global) minimiser. – user1551 Aug 01 '19 at 11:44
  • It seems to meet the necessary and sufficient conditions for optimality (i.e., both the first-order condition and the complementary slackness condition would appear to be met). But there must be something wrong with that argument, since it's easy to come up with trivial numerical examples where that $X$ does not attain the minimum trace. So I'll forget about it. – sunga Aug 02 '19 at 06:28