This study presents a face recognition model for construction site safety, integrating Support Vector Machine and Gabor algorithm with incremental learning and Multi-task Cascaded Convolutional Network. The model reduces Gabor’s computational complexity while enhancing feature selection and dimensionality reduction. Tested under three lighting conditions, it achieved 99.7% and 99.1% accuracy in high- and low-light environments, respectively. At a Signal-to-Noise Ratio of 15 dB, it maintained an accuracy of 86.2%, outperforming comparison models with accuracies of 70.6%, 68.3%, and 68.1%. The proposed model ensures efficient and low-computational cost face recognition in complex environments with incomplete facial data, significant lighting variations, and limited hardware. By improving recognition accuracy and adaptability, this model enhances safety management, worker protection, and supervision of construction personnel. The study provides a practical solution for construction site safety, addressing key challenges in real-world scenarios while maintaining high efficiency and accuracy.