We present a simple method for determining whether the noise in the pulse period history of X-ray pulsars can be described by a random walk process. This is done by calculating the mean difference in angular velocity between different observations as a function of the time interval between these observations. We apply the method to 10 X-ray pulsars for which more than 10 period determinations are available. For Vela X-1, we confirm earlier results that the pulse period behaviour is very well fitted by a random walk in period. In GX 301 - 2 the pulse period behaviour is also consistent with a random walk. Analysing the pulse period history of three X-ray pulsars with a Be companion, we find that a random walk is not consistent with the data. The pulse period history of the two best-studied X-ray pulsars that have an accretion disc, Cen X-3 and Her X-1, is not consistent with a random walk, but if a linear trend is removed the remaining variations are as expected for a random walk. In the systems showing random walk behaviour, the noise level is well correlated with the X-ray luminosity, and is not very different for the wind-accreting and disc-accreting systems. If we assume that in the wind-accreting sources the random walk results from accretion torques that change sign, as found in some turbulent accretion wake simulations in the literature, and that the specific angular momentum of the accreting matter is that of a Kepler orbit at the magnetospheric radius, we can derive a typical reversal time-scale from the observed noise level which is of the order of hours. This is in good agreement with the time-scales predicted by the simulations. In the disc-accreting systems, which are probably close to their equilibrium period, the noise is likely to be caused by short-term unbalanced fluctuations in either the spin-up or braking torques, and we derive lower limits on the magnitude of the required fluctuations. It is not clear yet why the disc systems should obey the same noise level-luminosity correlation as the wind-accreting systems.