Arbuscular mycorrhiza (AM) associations are among the most widespread interactions between plants and fungi. However, little is known about the AM distribution along altitudinal gradients in forest ecosystems, where the gradients are accompanied by shifts in climatic conditions and soil properties as well as the identity of host plant species. This study aimed to clarify AM fungal associations along an altitudinal gradient. Therefore, we determined the colonization rate and community composition of AM fungi at different elevations on Mount Ibuki, Japan. Root and soil samples were collected from Cryptomeria japonica at altitudes of 200, 500, and 800 m, Acer shirasawanum at an altitude of 1100 m, and Chrysanthemum japonicum at the summit of 1350 m. To evaluate AM fungal colonization, 15 root samples from each altitude were examined microscopically. These samples were identified by targeting V4 and V5 regions of a partial small subunit rRNA gene using Sanger sequencing. AM fungal colonization rates were significantly and positively correlated with altitude. The Arum and Paris types were observed in all plant species examined at each altitude. In total, 101 molecular operational taxonomic units belonging to two genera were identified: Glomus and Diversispora. No significant clustering among elevations was observed in the AM fungal community composition. Among environmental variables, soil temperature, and pH significantly affected AM fungal structure patterns. These results suggest that as altitude increases, plants require more tight mycorrhizal associations, which are selectively filtered by the surrounding environment.