In this paper we study matrix valued orthogonal polynomials of one variable associated with a compact connected Gelfand pair (G, K) of rank one, as a generalization of earlier work by Koornwinder [30] and subsequently by Koelink, van Pruijssen and Roman [28], [29] for the pair (SU(2) x SU(2), SU(2)), and by Grunbaum, Pacharoni and Tirao [13] for the pair (SU(3), U(2)). Our method is based on representation theory using an explicit determination of the relevant branching rules. Our matrix valued orthogonal polynomials have the Sturm-Liouville property of being eigenfunctions of a second order matrix valued linear differential operator coming from the Casimir operator, and in fact are eigenfunctions of a commutative algebra of matrix valued linear differential operators coming from the K-invariant elements in the universal enveloping algebra of the Lie algebra of G.