Q-learning for adaptive, load based routing.

The results reported in this paper concern the control problem of routing in packet switched internets. Using Q-learning an adaptive, distributed and autonomous routing strategy can be obtained. The objective of the Q-learner under study;is to balance the load such that average packet delivery time is optimised. If pure Q-learning is applied to routing each source has to learn the expected cost for sending a message via each of its neighbours for all destinations. Since Q-learning is basically a trial and error method packets have to be sent along non-optimal paths, which artificially increases the load. To reduce this effect and to speed up the learning a variant of Q-learning has been developed. Exploration and exploitation are partially decoupled such that stabilising features can be included in the Q-learning algorithm, to cope with instabilities and overhead that might be caused by the costly exploitation in search of alternative paths. Tn the paper the above statements will be justified mathematically and supported mathematically experiments.