The grinding and blending process significantly enhances the stability of the Buton rock asphalt (BRA) and asphalt blending system, but the influence mechanism of BRA’s mineral particle size distribution on the rheological performance of modified asphalt is still unclear. To this end, this study prepared BRA-modified asphalt samples with different particle size distributions to systematically analyze the effect of BRA particle size distribution on the rheological performance of BRA-modified asphalt and then conduct high-and low-temperature rheological tests. Meanwhile, the fluorescence microscopy test was adopted to analyze the phase structure of BRA-modified asphalt under different particle size distributions. The results show that D50, D90, and Dspan negatively correlate with high-temperature deformation resistance, and positively correlate with the low-temperature creep stiffness and creep rate. This suggests that reducing BRA’s particle size helps to improve the high-temperature stability and low-temperature crack resistance of modified asphalt. Additionally, reducing the particle size of inorganic minerals improves their dispersion state in asphalt, which makes the structure denser and thus enhances both high-temperature rutting resistance and low-temperature flexibility.