Comparison of a genetic algorithm and simulated annealing for automatic neural network ensemble development

Abstract

In the last decades ensemble learning has established itself as a valuable strategy within the computational intelligence modeling and machine learning community. Ensemble learning is a paradigm where multiple models combine in some way their decisions, or their learning algorithms, or different data to improve the prediction performance. Ensemble learning aims at improving the generalization ability and the reliability of the system. Key factors of ensemble systems are diversity, training and combining ensemble members to improve the ensemble system performance. Since there is no unified procedure to address all these issues, this work proposes and compares Genetic Algorithm and Simulated Annealing based approaches for the automatic development of Neural Network Ensembles for regression problems. The main contribution of this work is the development of optimization techniques that selects the best subset of models to be aggregated taking into account all the key factors of ensemble systems (e.g., diversity, training ensemble members and combination strategy). Experiments on two well-known data sets are reported to evaluate the effectiveness of the proposed methodologies. Results show that these outperform other approaches including Simple Bagging, Negative Correlation Learning (NCL), AdaBoost and GASEN in terms of generalization ability.