With the widespread application of deep learning technology in image analysis, visual-based data processing and action recognition have become current research hotspots. To improve the accuracy and real-time evaluation of the quality of aerobics actions, a new lightweight OpenPose network is developed to extract key points of athletes’ skeletons. At the same time, the adaptive multi-scale optimization is carried out on the spatiotemporal graph convolutional network to grasp the temporal characteristics of actions. Finally, a new aerobics action quality evaluation model is designed by integrating Siamese network. The accuracy of the proposed model reached 96.8%, which was 5.3% higher than existing methods. The highest action evaluation matching degree was 89.7%, the highest action improvement rate was 21.34%, the highest action similarity evaluation was 92.17%, and the shortest evaluation time was 0.71 seconds. Its efficiency and effectiveness in the detection and evaluation process are significantly higher than other advanced models. From this, the proposed method for evaluating the quality of aerobics actions has strong application potential while ensuring efficiency and accuracy, and can provide certain technical support for aerobics competitions and actions training.