To solve low toughness and high brittleness of epoxy asphalt under low-temperature conditions, in this study, polyether-type polyurethane (JMPU) and polyester-type polyurethane (JZPU) were employed to prepare modified epoxy asphalt. Firstly, the Brookfield viscosity test, infrared spectroscopy test, and fluorescence microscopy were utilized to characterize the curing process. The curing time and curing temperature of JMPU and JZPU modified epoxy asphalt were 3 h/140 °C and 4 h/150 °C, respectively. With styrene-butadiene-styrene block copolymer (SBS), a common modifier, as the control, the performance changes of epoxy asphalt before and after modification with different toughening agents were evaluated through mechanical tensile test, rheological property test, multiple stress creep recovery (MSCR) test, and storage stability test. The research findings indicate that the polyurethane system can effectively enhance the elongation at break, low-temperature performance, and storage stability of epoxy asphalt by forming an interpenetrating network structure (IPNs) with epoxy asphalt, while maintaining excellent high-temperature performance. The elongation at break of JMPU-modified epoxy asphalt and its low-temperature creep stiffness at ?12 °C can reach 233% and 169 MPa, respectively. Results of an aging test on the toughened and modified epoxy asphalt show that the incorporation of JMPU and JZPU effectively mitigates the adverse effects of aging on epoxy asphalt performance, evidenced by lower creep stiffness aging index and rutting factor aging index, as well as higher creep rate aging index. Therefore, the addition of the PU system as a toughening agent improves the toughness of epoxy asphalt and enhances its compatibility and anti-aging performance.