4

Given $y_1,\dots, y_n \in {\Bbb R}$, $w \in {\Bbb R}^d$, and $x_1,\dots, x_n \in {\Bbb R}^D$, how do we solve the following optimization problem

\begin{align} \min_A \,\, \sum_{i=1}^n \left( y_i - w^T A^T x_i \right)^2\\ \text{subject to} \qquad A^TA=I_d \end{align}

where $I_d$ is $d$ dimensional identity matrix, and $A \in {\Bbb R}^{D\times d}$, with $D \gg d$. My hope is that a solution to the above problem is possible, using only spectral arguments.

Rodrigo de Azevedo
  • 23,223
gmravi
  • 366

1 Answers1

0

According to the real singular value decomposition we can write $A=U I_{Dd} V$, where U is an orthogonal DxD-matrix, V is an orthogonal dxd-matrix and $I_{Dd}$ is the Dxd-matrix with 1s on the diagonal and zeros elsewhere. Now calculate $$w^TA^Tx_i=w^TU I_{Dd} Vx_i=(w^TU\oplus0)Vx_i=(w^T\oplus0)(U\oplus I)Vx_i.$$ Now we see that optimizing over $U$ and $V$ is the same as optimizing over a single orthogonal matrix, i.e. replacing $w^TA^Tx_i$ by $(w^T\oplus 0)O^Tx_i$ with $O$ orthogonal DxD. Hence the problem is equivalient to \begin{align} \min_e \sum_{i=1}^n (y_i-||w||e^Tx_i)^2 \end{align} optimized over all unit vectors $e\in \mathbb{R}^D$. Now you can use well known linear least squares results.

Christian Majenz
  • 26
  • I have the following few edits and questions. Firstly it should read as follows. $A^T=UI_{Dd}V$, where $U$ is a $d\times d$ orthogonal matrix, and $V$ is a $D\times D$ orthogonal matrix. The above calculation is then as follows. $w^TA^Tx_i=[w^TU,0]Vx_i=[w^T,0](U \oplus I)Vx_i=[w^T,0]O^Tx_i$, for some orthogonal matrix $O$ of size $D\times D$. However, I do not understand the final step where you claim that the problem is equivalent to the least squares problem, that you have shown above. It would be great if you could clarify that. – gmravi May 29 '14 at 22:53