In many cases, the dynamic behavior of a rotating system is strongly influenced by support parameters. The use of viscoelastic supports is a feasible solution for vibration control in rotating machinery. This article sets out to describe how to design an optimal viscoelastic supports for a flexible rotor, while the concept of the fractional order derivative has been applied to the construction of parametric models for viscoelastic supports. The motion equations for the flexible rotor model mounted on viscoelastic supports are derived and the approximately analytical solution is obtained. The optimal parameters of the fractional-order supports are analytically studied for the H∞ and H2 optimization criteria. The H∞ optimum parameters such as fractional coefficient and order are obtained based on the classical fixed-points theory to minimize the rotor amplitudes. The H∞ and H2 optimization parameters to minimize the total vibration energy of the flexible rotor over the whole-frequency range are also determined. The system optimization design can effectively improve the resonant vibration response as the results show. Consequently, the maximum rotor amplitude of the system can be reduced by more than 50% for both optimization procedures, while the optimum parameters are used. It could be concluded that the fractional viscoelastic support has superiority in vibration engineering, and fractional-order element could replace the traditional damper and spring simultaneously in some cases.