According to this answer, for $N$ observations and $C$ variables, the computational complexity of solving a linear regression problem $(X^\intercal X)^{-1}X^\intercal \mathbf y$ is $O(C^2N)$.
I am not good at analyzing complexity. If I were instead interested in weighted least squares, which has solution $(X^\intercal W X)^{-1}X^\intercal W \mathbf y$, what would its computational complexity be?
My attempt:
I am just following the pattern from this answer, which seems to say that multiplying three matrices $M \times N, N \times P, P \times M$ is $O(MNP + M^2 P)$.
This gets me:
- Multiplying $X^\intercal W X$ is $O(CN^2 + C^2N)$
- Multiplying $X^\intercal W \mathbf y$ is $O(CN^2 + C^2N)$
- Computing the LU of $X^\intercal W X$ and using it to compute the product $(X^\intercal W X)^{-1}X^\intercal W \mathbf y$ is $O(C^3)$
Since we assume $N > C$, $O(CN^2 + C^2N)$ dominates $O(C^3)$, so the final answer is $O(CN^2 + C^2N)$.
Is this right?
Thanks.
