Fire is one of the most serious hazards that structures could suffer during their service lives, which would impose devastating losses on lives and properties. Rapid prediction of thermal behavior and structural responses is essential for the assessment of structural fire safety. However, finite element methods (FEMs) typically come with high computational costs, and thus fail to meet the demand for fast computation. This study proposes a graph neural network (GNN)-based framework to predict structural thermo-mechanical responses. The geometric and physical information, e.g., fire area and boundary condition, is considered in the vertex embedding. Taking the fire position and heat flux as input, the proposed GNN model is able to provide temporal prediction on temperature, stress and displacement simultaneously. A physics-informed aggregate function is introduced in the message passing phase, enabling the heat conduction to be well captured by the message passing process. Moreover, new prediction schemes have been proposed to eliminate the effect of error accumulation in the rollout prediction. Two numerical examples are employed to verify the effectiveness of the proposed GNN-based framework. The results show that the proposed prediction model accurately predicts structural thermo-mechanical responses even for large-scale complex structures using a small number of training data, and runs an order of magnitude speedup compared to the FEM. The proposed GNN-based framework is of great potential for the application of real-time response prediction of large-scale realistic structures during fire emergencies.