Principal component analysis (PCA) is a linear dimensionality reduction technique. It reduces a multivariate dataset to a smaller set of constructed variables preserving as much information (as much variance) as possible. These variables, called principal components, are linear combinations of the input variables.
Questions tagged [principal-component-analysis]
143 questions
465
votes
4 answers
What is the intuitive relationship between SVD and PCA?
Singular value decomposition (SVD) and principal component analysis (PCA) are two eigenvalue methods used to reduce a high-dimensional data set into fewer dimensions while retaining important information. Online articles say that these methods are…
wickedchicken
- 4,761
13
votes
6 answers
Why eigenvectors with the highest eigenvalues maximize the variance in PCA?
I'm learning Principal Component Analysis (PCA) and came to know that eigenvectors of the covariance matrix of the data are the principal components, which maximizes the variance of the projected data. I understand the intuition behind why we need…
Kaushal28
- 705
12
votes
1 answer
Why is SVD on $X$ preferred to eigendecomposition of $XX^\top$ in PCA?
In this post J.M. has mentioned that ...
In fact, using the SVD to perform PCA makes much better sense numerically than forming the covariance matrix to begin with, since the formation of $XX^\top$ can cause loss of precision. This is detailed in…
S. P
- 303
10
votes
2 answers
Connection between PCA and linear regression
Is there a formal link between linear regression and PCA? The goal of PCA is to decompose a matrix into a linear combination of variables that contain most of the information in the matrix. Suppose for sake of argument that we're doing PCA on an…
user9576
- 355
9
votes
3 answers
How to prove PCA using induction?
In Deep Learning (Goodfellow, et al), the optimization objective of PCA is formulated as
$$D^* = \arg\min_D ||X - XDD^T||_F^2 \quad \text{s.t.} \quad D^T D=I$$
The book gives the proof of the $1$-dimension case, i.e.
$$\arg\min_{d} || X - X…
Lifu Huang
- 557
8
votes
1 answer
PCA produces sinusoids — what is the underlying cause?
Background
I'm analysing a data set of $M$ flow measurements (volume per time). The flows go from zero mL/s gradually to higher values and back to zero again, thus: their shapes ideally look like a Gaussian (or bell-shaped) curve. However, their…
Erik
- 605
7
votes
2 answers
Relationship between the singular value decomposition (SVD) and the principal component analysis (PCA). A radical result(?)
I was wondering if I could get a mathematical description of the relationship between the singular value decomposition (SVD) and the principal component analysis (PCA).
To be more specific I have some point which I don't understand very well, at…
Sergio Sarmiento
- 432
- 3
- 12
6
votes
0 answers
What is the term for two matrices sharing the same sets of singular vectors?
Consider the singular value decompositions $A=U_A\Sigma_AV_A^T$ and $B=U_B\Sigma_BV_B^T$. Is there a word that describes the relation between $A$ and $B$ when they have the same left and/or right singular vectors ($U_A=U_B$ and/or $V_A=V_B$)?
If we…
paperskilltrees
- 755
5
votes
4 answers
PCA and image compression
I have two questions related to principal component analysis (PCA):
How do you prove that the principal components matrix forms an orthonormal basis? Are the eigenvalues always orthogonal?
On the meaning of PCA. For my assignment, I have to…
5
votes
1 answer
Why do eigenvectors arise as the solution of PCA?
I have very limited knowledge of linear algebra and therefore I don't have an geometrical intuition behind PCA.
Why the eigen vectors (which are simply defined as vectors whose direction doesn't change after a linear transformation) are also the…
4
votes
2 answers
Fast PCA: how to compute and use the covariance of x
I'm trying to understand the paper Fast principal component analysis using
fixed-point algorithm by Alok Sharma and Kuldip K. Paliwal (1151–1155), and
especially what is said about $\Sigma_x$, the covariance of x.
But before being specific, let me…
ubitux
- 41
4
votes
1 answer
What is the principal components matrix in PCA with SVD?
Doing PCA on a matrix using SVD yields a result of three matrices, expressed as:
$$
M = U \Sigma V^T
$$
where $M$ is our initial data with zero mean.
If we want to make a plot of the two principal components we project the data onto principal…
Paul Hunter
- 141
4
votes
1 answer
Interpretation of PCA
I am wondering if there is a practical interpretation of a principal component analysis: Consider you have a data matrix $X\in\mathbb{R}^{N\times p}$ and you perform a principal component analysis where you typically receive certain directions…
JohnSmith
- 1,544
3
votes
1 answer
Why are the following equivalent formulations of the PCA optimization problem correct?
In these notes, the author formulates the PCA problem as follows. Given a matrix of data $X$ the PCA problem is:
$$\text{min}_Y \|Y - X\|_F \qquad \text{rank}(Y) = k$$
This reads to me as obviously correct.
But then the author claims that this…
Your neighbor Todorovich
- 8,472
3
votes
1 answer
quasi-PCA reconstruction of the matrix by orthogonal basis
let's say I have a "data" matrix $X$ of $N$ rows and $p$ cols with $N \gg p$. Now PCA with $L$ components can be formulated as $$X_L = argmin_{Y:rank(Y) = L} ||X- Y||^2_F $$, where Y is taken to be an $L$-dimensional approximation of $X$.
It's…
