In this paper, we present a new fractional theory of dynamics, i.e., the dynamics of generalized Hamiltonian system with fractional derivatives (fractional generalized Hamiltonian mechanics). Based on the definition of Riemann-Liouville fractional derivatives, the fractional generalized Hamiltonian equations are obtained, the gradient representation and second-order gradient representation of the fractional generalized Hamiltonian system are studied, and then the conditions on which the system can be considered as a gradient system and a second-order gradient system are given, respectively. By using the method and results of this paper, the conditions under which a fractional generalized Hamiltonian equation can be reduced to a generalized Hamiltonian equation, a fractional Hamiltonian equation and a Hamiltonian equation are given, respectively, and then the existing conditions and their form of gradient equation and second-order gradient equation are investigated. Finally, an example of a fractional dynamical system is given to illustrate the method and results of the application.