Trading Bias for Expressivity in Artificial Learning

Bias, arising from inductive assumptions, is necessary for successful artificial learning, allowing algorithms to generalize beyond training data and out-perform random guessing. We explore how bias relates to algorithm flexibility (expressivity). Expressive algorithms alter their outputs as training data changes, allowing them to adapt to changing situations. Using a measure of algorithm flexibility rooted in the information-theoretic concept of entropy, we examine the trade-off between bias and expressivity, showing that while highly biased algorithms may outperform uniform random sampling, they cannot also be highly expressive. Conversely, maximally expressive algorithms necessarily have performance no better than uniform random guessing. We establish that necessary trade-offs exist in trying to design flexible yet strongly performing learning systems.