Traditional manual inspection methods for concrete structures exhibit inherent limitations in complex construction environments, including high operational costs, subjective assessment biases, and spatial localization challenges. To mitigate these limitations, this study developed a mobile sensing system that combines deep learning and ultra-wideband (UWB) technologies to enable real-time defect detection and three-dimensional mapping within a virtual web-based building model. The contributions of this study lie in three aspects: (1) A lightweight Faster Real-Time Detection Transformer (Faster-RTDETR) deep learning model, optimized for multi-defect detection. This model achieves a mean average precision (mAP) of 89.3%, a significant improvement over baseline models, while reducing the number of parameters by 39.3%. (2) A mobile sensing hardware system, consisting of a portable helmet platform, an image acquisition module, an image processing module, and a localization module, was developed for real-time defect detection and multi-directional and multi-angled inspections of target areas in construction sites; (3) A UWB-based spatial localization method that maps construction quality defects to a three-dimensional BIM platform for visualization and efficient management. Field tests on high-rise concrete structures validated the system’s high accuracy in detecting cracks, spalling, and voids, thereby enabling real-time construction quality assessment. This study presents a novel framework for automated, intelligent quality inspection of concrete structures under construction, which significantly enhances real-time defect detection capabilities and spatial defect localization efficiency in a BIM platform.