The diversification of aggregate sources in pavement engineering has led to significant differences in aggregate properties, thereby affecting the adhesion performance between aggregates and asphalt. Accurately evaluating and distinguishing the adhesion performance between multi-source aggregates and asphalt are important prerequisites for ensuring the performance of asphalt pavement. A method for evaluating multi-source aggregates and modified asphalt based on rotary bottle tests and image processing was proposed. Rotary bottle tests were conducted on multi-source aggregates coated with modified asphalt, and the adhesion performance was quantitatively evaluated through the asphalt spalling rate obtained from image processing. Furthermore, the influence of factors such as immersion time and asphalt aging was analyzed. The evaluation results indicate that this method can clearly distinguish the differences in adhesion performance of multi-source aggregates, which has certain practical and engineering significance.