Visual Autonomy via 2D Matching in Rendered 3D Models

As they decrease in price and increase in fidelity, visually-textured 3D models offer a foundation for robotic spatial reasoning that can support a huge variety of platforms and tasks. This work investigates the capabilities, strengths, and drawbacks of a new sensor, the Matterport 3D camera, in the context of several robot applications. By using hierarchical 2D matching into a database of images rendered from a visually-textured 3D model, this work demonstrates that - when similar cameras are used - 2D matching into visually-textured 3D maps yields excellent performance on both global-localization and local-servoing tasks. When the 2D-matching spans very different camera transforms, however, we show that performance drops significantly. To handle this situation, we propose and prototype a map-alignment phase, in which several visual representations of the same spatial environment overlap: one to support the image-matching needed for visual localization, and the other carrying a global coordinate system needed for task accomplishment, e.g., point-to-point positioning.