A model approach is developed to study intermediate steps and transient structures in a course of the membrane self-assembly. The approach is based on investigation of mixed lipid/protein-detergent systems capable of the temperature induced transformation from a solubilized micellar state to closed membrane vesicles. We performed a theoretical analysis of self-assembling molecular structures formed in binary mixtures of dimyristoylphosphatidylcholine (DMPC) and sodium cholate (NaC). The theoretical model is based on the Helfrich theory of curvature elasticity, which relates geometrical shapes of the structures to their free energy in the Ginzburg-Landau approximation. The driving force for the shape transformation is spontaneous curvature of amphiphilic aggregates which is nonlinearly dependent on the lipid/detergent composition. An analysis of the free energy in the regular solution approximation shows that the formation of mixed structures of different shapes (discoidal micelles, rod-like micelles, multilayer membrane structures and vesicles) is possible in a certain range of detergent/lipid ratios. A transition from the flat discoidal micelles to the rod-like cylindrical micelles is induced by curvature instabilities resulting from acyl chain melting and insertion of detergent molecules into the lipid phase. Nonideal mixing of the NaC and DMPC molecules results in formation of nonideal cylindrical aggregates with elliptical cross section. Further dissolution of NaC molecules in DMPC may be accompanied with a change of their orientation in the lipid phase and leads to temperature-induced curvature instabilities in the highly curved cylindrical geometry. As a result the rod-like micelles fuse into less curved bilayer structures which transform eventually to the unilamellar and multilamellar membrane vesicles. The theoretical analysis performed shows that a sequence of shape transformations in the DMPC/NaC mixed systems is determined by the synergism of four major factors: detergent/lipid ratio, temperature (acyl chain melting), DMPC and NaC mixing, and reorientation of NaC molecules in mixed aggregates.