Planetary systems can survive stellar evolution, as is clear from the atmospheric metal pollution and circumstellar dusty disks of single white dwarfs(1,2). Recent observations show that 1-4% of single white dwarfs are accompanied by dusty disks(3-6), while the occurrence rate of metal pollution is about 25-50%(1,7,8). Dusty disks and metal pollution have been associated with accretion of remanent planetary systems around white dwarfs(1,9), yet the relationship between these two phenomena is still unclear. Here, we suggest an evolutionary scenario to link the dusty disk and metal pollution. By analysing a sample of metal-polluted white dwarfs, we find that the mass accretion rate onto the white dwarf generally follows a broken power-law decay, which matches well with the theoretical prediction, assuming that dust accretion is primarily driven by Poynting-Robertson drag(10) and the dust source is primarily delivered via dynamically falling asteroids perturbed by a Jovian planet(11,12). The presence of disks is mainly at the early stage (t(cool) similar to 0.1-0.7 Gyr) of the whole process of metal pollution, which is detectable until similar to 8 Gyr, naturally explaining the fraction (similar to 2-16%) of metal-polluted white dwarfs with dusty disks. The success of this scenario also implies that the configuration of an asteroid belt with an outer gas giant might be common around stars of several solar masses.