This paper analyses the convexity of the problem formulations for topological optimization of compliant mechanisms. In this context, a ground structure is assumed to consist of membranes with variable thickness or trusses with variable cross-sectional area. It is proven that the objective function of maximum mechanical or geometrical advantage may not be convex even though the constraints for the problem formulations give rise to a convex set. The constraints include volumetric constraint and a bound on input displacement. The nonconvex objective functions are shown analytically with two examples using truss elements. Also, it is illustrated numerically with an example using bilinear quadrilateral elements. It is concluded that nonconvex objective functions also attribute to difficulties in arriving at the globally optimal topology of compliant mechanisms, in addition to the number of design variables and constraints. As the problem is more complicated, the solution of the problem requires robust and reliable optimization techniques.