Unveiling rare patterns : enhancing interpretability and discovering unexpected insights in data

Abstract

In data analysis, rare pattern mining is essential for uncovering valuable insights by identifying uncommon patterns that often escape traditional methods. Despite notable advancements in deep learning, challenges related to explainability and interpretability persist, particularly when it comes to identifying and understanding rare, but impactful patterns. Rare pattern mining, especially association rule mining, offers an interpretable approach that provides insights that are crucial for informed decision-making. This doctoral dissertation addresses key challenges in rare pattern mining through four main contributions: (1) developing an efficient method for discovering rare patterns, (2) discovering the concise representation of rare patterns to reduce redundancy, (3) unveiling interesting patterns by filtering out irrelevant and noisy patterns, and (4) demonstrating practical applicability through a case study focused on interpretability. First, we introduce a novel depth-first search approach to overcome the limitations of traditional methods, achieving substantial improvements in speed and memory efficiency, particularly in sparse datasets, and thus establish a new standard for rare pattern extraction. To address redundant pattern generation, we propose a method for identifying maximal rare patterns, providing a concise output that enhances analysis efficiency. In addition, we introduce a model that isolates interesting patterns by effectively filtering out irrelevant and noisy data, ensuring that only impactful patterns are highlighted, thereby enhancing interpretability and reducing information overload. The practical implications of these contributions are demonstrated in a healthcare case study focused on heart disease, where interpretability and transparency are essential. By analyzing patient data, our methods not only reveal critical risk factors but also identify vulnerability patterns in asymptomatic individuals, enabling early intervention and improved health outcomes. This case study underscores the value of model transparency and interpretability, particularly in high-stakes applications. Through these contributions, this thesis makes significant advances in the efficiency, relevance, and interpretability of rare pattern mining. The proposed methods provide robust, actionable insights across various domains, providing a foundation for more effective data-driven decision-making.