The frequency-domain spectrum (FDS) method can effectively evaluate the aging state of crosslinked polyethylene (XLPE) cable insulation. To more accurately achieve a quantitative analysis of information in the FDS, this article combines the concepts of fractional element (Cap-Resistor) and fractional calculus, inspired by the Debye model and the multibranch improvement method of the Cole-Cole (C-C) model, establishes an improved fractional Zener model based on the existing Cap-Resistor model, and combines the improved fractional Zener model with the elite genetic algorithm (EGA) to propose a new method for analyzing the relaxation polarization information of XLPE dielectric. Six sets of XLPE cable samples with different aging times (including a validation set) are prepared at an aging temperature of 150 degrees C, and the FDS of the thermal aging samples is measured. Then, the FDS is fit using a modified fractional Zener model: first, the least squares method is used to establish the optimized objective function, the characteristic parameters are extracted and preferentially selected using EGA for analysis, and the fitting equations between aging time and characteristic parameters are established, and the validation results manifest that the error of the characteristic parameters on the aging time characterization of XLPE cable insulation samples is 0.18465 %, which is excellent. The evaluation method's accuracy has been validated. This article introduces a feasible approach for analyzing relaxation polarization information in XLPE dielectric using an improved fractional Zener model and EGA.