Is there a fast way to invert $V^T D V$ where $V\in\mathbb{R}^{m\times n}$ is orthonormal, $D\in\mathbb{R}^{m\times m}\succ 0$ diagonal, and $m>n$?

Question

Is there a fast way to invert $V^T D V$ where $V\in\mathbb{R}^{m\times n}$ contains orthonormal columns, $D\in\mathbb{R}^{m\times m}$ is positive definite and diagonal, and $m>n$? I'd prefer to avoid factorizing the full matrix, $V^T D V$. To be clear, $V^TD^{-1}V$ does not work because $V$ contains more rows than columns.