FlyView: a bio-informed optical flow truth dataset for visual navigation using panoramic stereo vision

Flying at speed through complex environments is a difficult task that has been performed successfully by insects since the Carboniferous [1], but remains a challenge for robotic and autonomous systems. Insects navigate the world using optical flow sensed by their compound eyes, which they process using a deep neural network implemented on hardware weighing just a few milligrams. Deploying an insect-inspired network architecture in computer vision could therefore enable more efficient and effective ways of estimating structure and self-motion using optical flow. Training a bio-informed deep network to implement these tasks requires biologically relevant training, test, and validation data. To this end, we introduce FlyView1, a novel bio-informed truth dataset for visual navigation. This simulated dataset is rendered using open source 3D scenes in which the agent's position is known at every frame, and is accompanied by truth data on depth, self-motion, and motion flow. This dataset comprising 42,475 frames has several key features that are missing from existing optical flow datasets, including: (i) panoramic camera images, with a monocular and binocular field of view matched to that of a fly's compound eyes; (ii) dynamically meaningful self-motion, modelled on motion primitives or the 3D trajectories of drones and flies; and (iii) complex natural and indoor environments, including reflective surfaces, fog, and clouds.