Fabric defect detection (DD) is critical in ensuring high-quality textile production, as defects can significantly impact aesthetic appeal and market value. Manual inspection methods are often inefficient and prone to inconsistency, motivating the development of intelligent, automated systems. Despite advancements in computer vision and deep learning (DL), many current models, such as rule-based methods involving thresholding and edge detection, still face challenges in accurately detecting subtle or complex defects under varied lighting and texture conditions. This research addresses these gaps by proposing a novel, DL-driven model named Binary Gannet Optimizer-driven Gate Adjusted Long Short-Term Memory Network (BGO-GALSTM-Net) for robust and precise defect detection and classification in textiles. A comprehensive dataset, ensuring diversity in defect types such as stains, holes, and misweaves, is used to train and evaluate the proposed model. Data preprocessing involves contrast enhancement and edge-preserving smoothing to improve defect visibility. Feature extraction uses a residual network (ResNet), allowing the model to focus on intricate defect regions. The proposed BGO-GALSTM-Net integrates temporal memory capabilities of LSTM with adaptive gating, while the Binary Gannet Optimizer fine-tunes model parameters for optimal performance. This architecture effectively processes the input image data, learns spatiotemporal patterns, and classifies defect types with high accuracy. Experimental results demonstrate superior accuracy (97.4%) along with higher F1-score, precision, and recall across various fabric types, outperforming traditional approaches. The proposed framework provides a reliable, scalable solution for real-time textile quality control, enabling manufacturers to reduce waste and maintain stringent product standards.