Color and the Cone Mosaic

Vision is limited by the measurements taken by the cone photoreceptors. To provide useful perceptual representations, the brain must go beyond the measurements and make inferences about the scene being viewed. This article considers the first stages of spatiochromatic vision. We show how spatial and chromatic information become intertwined by the optics of the eye and because of the structure of the retinal cone mosaic, and we consider the consequent implications for perception. Because there is at most one cone at each retinal location, the standard treatment of human trichromacy does not apply at fine spatial scales. Rather, trichromacy results from a perceptual inference based on measurements from cones of different classes at different locations. Our treatment emphasizes linking physics, biology, and computation with the goal of providing a framework for a larger understanding of how the brain interprets photoreceptor excitations to see objects and their properties.