In today’s era of widespread education, there is a complex correlation between students’ academic performance and their behavior. In order to find the potential relationship between students’ learning behavior and academic performance from complex teaching data and provide better teaching solutions for educators, this paper used the Apriori algorithm to mine association rules in educational data. This study collected learning data from 5000 students in their first to third year of high school from a key higher education institution and a non-key higher education institution. Through data preprocessing and transformation, the optimized Apriori algorithm was used for frequent itemset mining, and four main learning behavior patterns were identifiedonline discussion, completing assignments on time, classroom participation, and additional reading. The experimental results show that the optimized algorithm reduces the execution time by 32% from 12.5 seconds before optimization to 8.5 seconds after optimization when processing the same amount of data. The performance of the optimized algorithm is stable when processing large-scale data. As the amount of data increases, the running time of the algorithm steadily increases from 2.4 seconds to 15.0 seconds, and the number of frequent itemsets gradually increases from 48 to 270. Research has found a significant correlation between frequent participation in online discussions and timely completion of assignments with student performance, particularly among students who attend regularly and have a higher Grade Point Average (GPA), with support and confidence levels of 0.22 and 0.85, respectively. The results of this study have been successfully applied to the development of targeted tutoring and personalized learning plans in a certain higher education institution, which significantly improved students’ academic performance and classroom participation.