The color associated with an object in machine vision images is not constant; under varying illuminating and viewing conditions (such as in outdoor images), the perceived color of an object can vary significantly, thus making color-based recognition difficult. Existing methods in color-based recognition have been applied mostly to indoor and/or constrained imagery, but not to realistic outdoor data. This work analyzes the variation of object color in outdoor images with respect to existing models of daylight illumination and surface reflectance. Two approaches for color recognition are then proposed: the first develops context-based model of daylight illumination and hybrid surface reflectance, and predicts the color of objects based on scene context. The second method shows that object color can be nonparametrically "learned" through classification methods such as Neural Networks and Multivariate Decision Trees. The methods have been successfully tested in domains such as road/highway scenes, off-road navigation and military target detection.