Information Theory in Multi-Label Feature Selection: An Analytical Review

In the context of multi-label learning, feature selection (MLFS) is a key process for handling high-dimensional datasets, aiming to retain the most informative features while preserving inter-label relationships.  This study presents an extensive overview of state-of-the-art MLFS approaches founded on principles from information theory.  The paper first introduces the fundamental concepts of information theory, then provides a detailed review of representative MLFS methods along with their theoretical background.  Performance assessments are carried out on real-world multi-label datasets, allowing us to highlight the advantages and shortcomings of each method.  For comparison, we employ widely used evaluation metrics such as Hamming Loss, Accuracy, Label Ranking Loss, and F1-score.  We also outline future research perspectives, including the design of a new feature relevance criterion that integrates label importance weighting and redundancy reduction based on label dependencies, with the aim of enhancing both feature selection and multi-label classification accuracy.