My first feeling when seeing PCA were very mixed: this is some magic, it’s awesome, it’s stupid, it can’t do any good, it’s not readable, it will do great stuff, it won’t help anything… So, I needed to understand it better and run some small experiment.

irst assumption about PCA is that it finds linear relations… Or rather, linear transformations of data that do not lose much information… ehmm… it’s math, nobody likes math, it’s ugly, it’s nerdy, it’s sexy, but it’s beyond the means of average humans. It’s like a trillion dollars bill right in front of you, that you will never get – it’s beyond annoying. So, let’s leave the details to mathematicians and praise them and their matrices, eighenvectors and all their black magic… and keep them away at the same time. Anyway – PCA doesn’t work with non-linear relations between different features of the data. It just doesn’t.

So, I was wondering – if x has an exponential correlation with y and for linear algebra to work with it one can just replace y with exp(y) to linearize correlation… what if somehow PCA could capture a strong relation between x and y and then we could run PCA iteratively on [x, y, PCA1(x,y), PCA2(x,y)]? Could it magically find out that y is (x1 + x2 + pca1 + pca2) – yeah, now I see how stupid it was. I would need to multiply variables somehow, not add them to data. That is still something that can work, and I could explore – to automatically try various paths and detect if squaring a variable makes it more helpful or less… anyway, that is the song of the distant future.

For now, I played a simple experiment. I had three features of data, m, v, and e, where e = m*sqrt(v). Just for fun, I ran PCA on this dataset, and it was so absolutely accurate. Like 99.x% of data variance can is captured by pca1, which cares almost only about e value. Yeah… e is a composition of m and v, but there is an infinite amount of solutions for each e, so it didn’t feel right.

Therefore, I tried the other thing. I just recreated original data using PCA component vectors from pca1 and pca2. Over 99% of variance was covered, so reconstructed data should be close to the original. Yeah…

The conclusion is – whenever using PCA, try reconstructing the data to test if PCA is applicable to your data. Don’t trust its accuracy unless proven. It lies to you, with premeditation, it wants you to be wrong, to get fired and homeless… and you still can’t live without it! PCA is a woman, an extremely beautiful woman that will cheat on you the very second you give her something she doesn’t like. Just like my daughter.