The work aims to enhance the loading efficiency of goods and meet the stacking requirements of customized door and window materials, so as to improve the space utilization of pallet stacking. Firstly, a mathematical model for stacking doors and windows materials was constructed based on practical situations. Subsequently, in allusion to the issue of spatial utilization of door and window materials within stacking pallets, a stacking strategy employing empty maximal spaces was devised. An ant colony algorithm was designed to search for the optimal stacking position and orientation for door and window materials. Finally, a comparison of spatial utilization rates was conducted with traditional empirical algorithms. The results indicated that, under the same order sample size, the ant colony algorithm proposed in this study could improve the spatial utilization rate of stacks by 8.82% compared with traditional empirical algorithms. Additionally, the obtained stacking configurations exhibited lower stack heights and greater stability. The mathematical model proposed in this study provides a basis for the overall optimization of stacking configurations for online stacking of door and window materials.