Bridge damage detection is crucial for ensuring the safety and integrity of the bridge structure. Traditional methods for damage detection often rely on manual inspections or sensor-based measurements, which can be time-consuming and costly. In recent years, computer vision techniques have shown promise in bridge displacement measurement and damage detection. The objective of this study is to extract reliable features from displacement measured with computer vision-based method that are sensitive to structural condition change while robust to the variation of operational condition. In particular, thisresearch paper presents a novel approach for bridge damage detection using an indicator defined based on the transverse influence ratio (DTIR) from computer vision-based displacement measurements. The proposed method utilizes computer vision algorithms to extract bridge girder displacement responses under moving load. The DTIR indicator, defined as the vehicle-induced bridge quasi-static displacement ratio between two adjacent girders, is extracted as the damage-sensitive feature. Theoretical derivation proves that DTIR indicator is only related to the structural condition and the transverse position of a vehicle over the deck, while independent of the variation of vehicle weight and speed. To validate the effectiveness of the proposed method, a series of drive-by experiments were performed on a multi-girder beam bridge with different structural conditions. The results demonstrated the capability of the proposed approach in accurately detecting the occurrence and possible location of structural damage. Furthermore, the paper discusses the advantages and limitations of the DTIR indicator for bridge damage detection, as well as how to generalize the proposed method to bridges with more than two traffic lanes. In conclusion, the proposed method offers a promising solution for low cost, easy deployable and scalable health monitoring solution for bridges under operating conditions.