To improve the comprehensive performance of desulfurized rubber asphalt, the asphalt was modified by using desulfurized crumb rubber and polyphosphoric acid. Based on the response surface method, the preparation parameters of the composite modified asphalt were optimized. In addition, four types of modified asphalt were prepared by using ordinary rubber, desulfurized rubber, composite polyphosphoric acid/ordinary rubber, and composite polyphosphoric acid/desulfurized crumb rubber. Their performance was compared. The results show that compared with ordinary rubber asphalt, the asphalt modified by desulfurized rubber has lower viscosity, significantly lower mixing and compaction temperatures, and more stable compatibility between rubber powder and asphalt in the system, which facilitates the preparation, storage, transportation, and construction process of rubber asphalt. In terms of road performance, the high-temperature deformation resistance of asphalt modified by polyphosphoric acid/desulfurized crumb rubber shows a level similar to that of ordinary rubber asphalt, and its creep ability is more excellent in low-temperature environments. Among the four types of modified asphalt, the asphalt modified by polyphosphoric acid/desulfurized crumb rubber has more obvious performance advantages and is suitable for promotion and application.