FastRF accuracy and speed benchmarks
We tested on a machine with Core i7 4770S, 16 GB RAM, JVM 1.8, Weka 3.8 running Windows 10 (10.0 Enterprise Edition 64 bits). The forest size has been set to 500 trees. For each dataset for each algorithm, 5 runs of 10 fold cross-validation have been executed and averages reported.
The algorithms were tested in 45 synthetic datasets generated using the RDG1 Weka generator, 45 synthetic datasets generated using the BayesNetwork Weka generator and 33 real datasets, most of them from the UCI repository. Almost all the datasets have either a considerable number of instances or features (or both), since we were interested in performance of FastRF 2.0 beta on datasets that take a lot of time to train a model. The main conclusions on the 33 real datasets.
The speedup (Time Weka / Time FRF_2.0 beta) in real datasets is 1.77-fold for datasets with less than 1000 attributes and 3.54-fold for the datasets with more than 1000 attributes.
The benchmarks suggest that FRF_2.0 beta is faster than its predecessor (FRF_0.99) and the RF in Weka 3.8. The biggest speed-up is achieved in datasets where there are lots of attributes, which tend to be the datasets that also the ones that take more time to train. These are good news, because execution time is improved where we need it most.
| Name | Time Weka (ms) | Time Weka / Time FRF_2.0b |
|---|---|---|
| gisette_train.arff | 113693 | 2.39 |
| nomao.arff | 179149 | 1.78 |
| amazon-commerce-reviews.arff | 246351 | 7.43 |
| p53_mutants_train.arff | 307339 | 1.60 |
| skin-segmentation.arff | 460661 | 2.38 |
| dorothea_train.arff | 497086 | 4.47 |
The three (of 33) datasets where Fast RF 2.0 beta doesn't improve speed over Weka RF are:
| Name | Time Weka / Time FRF_2.0b |
|---|---|
| kdd_ipums_la_98-small.arff | 1.03 |
| kdd_ipums_la_99-small.arff | 1.04 |
| nursery.arff | 0.92 |
That is likely because these datasets contain multi-categorical attributes with lots of different values in each attribute. The one vs all split in FastRF 2.0 beta on categorical features is slower than the multifurcating split in Weka RF; this might be changed in FastRF in future release.
y axis shows fold-difference in time elapsed.
The accuracy of FastRF compared to Weka is overall similar, with some differences across datasets. Of note, we see a large increase in crossvalidation accuracy (as % instances correctly classified) in amazon-commerce-reviews.arff (+14.53 percentage points [pp]) and a decrease in ohscal.wc.arff (-5.26 pp). However, when the accuracy is estimated via crossvalidation area under the ROC curve (AUC) score, these differences are less evident: amazon-commerce-reviews.arff has +0.025 AUC and ohscal.wc.arff has -0.006 AUC. We currently interpret this as FastRF sometimes yielding a different distribution of the class probability estimates (votes in the forest) than the Weka RF. Therefore, applying a fixed threshold to decide on a class might give different results, depending on the FastRF 2.0 beta versus Weka RF.
The average accuracy difference across the 33 datasets is close to zero (+0.245 pp in favor of FastRF 2.0 beta), and the median difference is 0.008 pp. The average AUC difference is +0.0003 and median AUC difference is only +3e-5, all estimated in crossvalidation (five runs of 10-fold xval). Therefore we conclude that the main changes in the RF algorithm that were introduced in FastRF 2.0 -- selection of feature subsets for individual trees -- does not appear to have an overall effect on accuracy.
Here is the link to the folder with the results in an Excel document:
https://github.com/jordipiqueselles/FastRandomForest/tree/master/results