In this paper, a novel fractional-order no-equilibrium chaotic system with hidden attractor is presented. The dynamical characteristics of the fractional-order system are analyzed by the phase diagram, Lyapunov exponents, bifurcation diagram, complexity, and attractor basin. Based on the above analysis, an image encryption scheme performs discrete cosine transform on the R, G, and B channels of the original color image to get the corresponding sparse coefficient matrices. Then, the measurement matrix generated by the Hadamard matrix and the chaotic pseudo-random sequence is used to compress and perceive the sparse coefficient matrices. In addition, the row and column scrambling and GF (257) domain diffusion algorithm are performed on the compressed pixel matrix to obtain the final cipher image. Experimental results and performance analysis display that the scheme has high compressibility and security. Even if the compression rate is 0.25, the calculated PSNR values are around 30. In addition, the chi(2)-value of the encrypted Lena image is 248.2824, and the algorithm has passed the UACI and NPCR tests and can resist differential attacks. Therefore, the proposed algorithm is effectively.