{"id":173,"date":"2024-03-11T22:26:58","date_gmt":"2024-03-11T22:26:58","guid":{"rendered":"https:\/\/aulendil.net\/hallucinations\/?p=173"},"modified":"2024-03-11T22:26:58","modified_gmt":"2024-03-11T22:26:58","slug":"civilizng-the-random-forest","status":"publish","type":"post","link":"https:\/\/aulendil.net\/hallucinations\/civilizng-the-random-forest\/","title":{"rendered":"Civilizng the Random Forest"},"content":{"rendered":"\n
\"\"<\/figure>\n\n\n\n

In my previous post<\/a>, I explained that I was pondering the rather uneven distribution of features among the trees in a random forest. It didn’t feel right to just verify whether I could achieve a more uniform distribution. I felt compelled to explore how it functions with this wild notion of democracy. Yes, the random forest is akin to a mathematical proof that democracy works\u2014a bunch of ignorants casting votes for the final result. And time and again, it’s shown to be superior to relying on a single, even wisest tree.<\/p>\n\n\n\n

It all sounded easy. It always does. Just create a bunch of trees, compile the results, summarize those results, and incorporate a more uniform distribution. It always sounds easy until I actually try to do something.<\/p>\n\n\n\n

Long ago, in a galaxy far, far away, when the world seemed full of promise, I built this class for ARandomForest with universal distribution. I even created a separate script that could run some tests. I used the simple IRIS dataset\u2014not that I know what it is, probably some flowers, likely remnants of the good old hippie era\u2014and ran it through these tests. After rectifying a few bugs that prevented my code from running, I obtained the first results. The accuracy for the professional scikit-learn library was perfect. The accuracy for my code was also perfect, albeit in both cases without cross-validation, but it was a first attempt just to debug the code and check if it runs.<\/p>\n\n\n\n

Okay, not bad; it runs. It’s just that the data is so straightforward that even a deaf and blind dwarf could figure out how to sort it. So, I tried something more complex – data about wines and their quality, available somewhere on the University of California, Irvine page. And the results were devastating. My algorithm was clearly inferior to scikit-learn’s. So my idea was nonsense, I’m no good, I will never do anything sensible… But it didn’t give me peace: how could it be that making a more uniform selection of features resulted in worse outcomes? I speculated that, perhaps by sheer luck, the random forest from those ugly scientists at scikit-learn was fortunate to assign the best features… So I ran the test over and over with different seeds for the random generator… and it was always better. It didn’t feel right. They may be superior, but they must be employing some unfair black magic, I’ll catch them.<\/p>\n\n\n\n

I wanted to compare apples to apples, so I needed to have the same algorithm\u2014one with and another without the trick. Thus, I made a copy of my naive random forest classifier and, in this version, I didn’t use the more universal distribution but just randomly selected features as described in lectures. And I ran the tests comparing three solutions. Here are the results:<\/p>\n\n\n\n