Student behavior recognition is of great significance in the intelligent classroom environment for improving teaching quality, achieving personalized learning, and optimizing classroom management. However, the accuracy and real-time performance of existing technologies in complex scenarios still have limitations. To improve the accuracy and real-time performance of student behavior recognition, the study improved the eighth-generation model of “You Only Look Once” based on the squeezing—incentive attention mechanism, feature pyramid network structure, and anchor box mechanism, significantly enhancing the accuracy and robustness of student behavior recognition. The SE attention mechanism improves the efficiency of feature extraction by enhancing the dependency relationship between feature channels. The FPN structure enhances the multi-scale feature fusion ability by fusing features at different levels. Meanwhile, combined with the DeepSort real-time multiobject tracking algorithm based on deep learning, the problems of identity switching and trajectory loss in object tracking have been effectively solved, continuous tracking of students’ behaviors has been achieved, and the real-time performance has been significantly improved. In the experimental results, the average accuracies of the improved You Only Look Once eighthgeneration model on the SCB-dataset3 and CampusGuard datasets reached 82.3% and 83.0%, respectively, which was significantly better than that of the control models. The multi-target tracking accuracy of the DeepSort algorithm is 84.2% and 83.7% respectively, and it also performs well in terms of robustness and real-time performance. The results show that the improved You Only Look Once eighth-generation model and the real-time multi-object tracking algorithm based on deep learning can effectively improve the performance of student behavior recognition and tracking, providing strong technical support for teaching management and personalized learning in the intelligent classroom environment.