This study proposes a hybrid Physical Education (PE) teaching quality evaluation system that integrates a K-medoids clustering algorithm with an enhanced CNN-LSTM neural network. Traditional evaluation methods are often subjective and inconsistent, failing to capture the complex, time-varying nature of student behavior in PE classes. The proposed model preprocesses historical classroom data through feature correlation analysis and PCA-based dimensionality reduction, followed by K-medoids clustering to improve data structure and training efficiency. It then takes Ensemble Empirical Mode Decomposition (EEMD) to enhance the input representation of the LSTM model. Experimental results demonstrated that the improved CNN-LSTM achieved an F1 value of 0.98 and an RMSE of 0.11 with 1000 training samples, significantly outperforming baseline models including CNN, LSTM, and GRNN. The model showed peak accuracy of 97.6% during 10: 00–12:00 time slots, with an average recall of 90.4% across-varied student states. User evaluation by PE teachers indicated an average satisfaction score of 93.7. This model has been proven to be effective in handling nonlinear and non-stationary classroom data and can achieve real-time, objective, and personalized evaluation. Future improvements include expanding the dataset with classroom sensor data and incorporating cognitive and emotional engagement indicators.