It is well-known that a point T is an element of cv(N) in the (unprojectivized) Culler-Vogtmann Outer space cv(N) is uniquely determined by its translation length function parallel to center dot parallel to(T): F-N -> R. A subset S of a free group F-N is called spectrally rigid if, whenever T,T' is an element of cv(N) are such that parallel to g parallel to(T) = parallel to g parallel to(T') for every g is an element of S then T = T' in cv(N). By contrast to the similar questions for the Teichmuller space, it is known that for N >= 2 there does not exist a finite spectrally rigid subset of F-N. In this paper we prove that for N >= 3 if H <= Aut(F-N) is a subgroup that projects to a nontrivial normal subgroup in Out(F-N) then the H-orbit of an arbitrary nontrivial element g is an element of F-N is spectrally rigid. We also establish a similar statement for F-2 = F(a, b), provided that g is an element of F-2 is not conjugate to a power of [a, b].