Let G be a finite group. The prime graph P(G) of G is defined as follows. The vertices of P(G) are the primes dividing the order of G and two distinct vertices p, p' are joined by an edge if there is an element in G of order pp'. Let p be a prime number. In [4], the authors determined the structure of finite groups with the same element orders as PGL(2,p), and it is proved that there are infinitely many nonisomorphic finite groups with the same element orders as PGL(2,p). Therefore there are infinitely many nonisomorphic finite groups with the same prime graph as PGL(2,p). We know that PGL(2,p) has a unique nonabelian composition factor which is isomorphic to PSL(2,p). Let p be a prime number which is not a Mersenne or Fermat prime and p not equal 11, 19. In this paper we determine the structure of finite groups with the same prime graph as PGL(2,p) and as the main result we prove that if G is a finite group such that Gamma(G) = Gamma(PGL(2,p)) and p not equal 13, then G has a unique nonabelian composition factor which is isomorphic to PSL(2,p) and if p = 13, then G has a unique nonabelian composition factor which is isomorphic to PSL(2, 13) or PSL(2, 27).