Vector-Based Semantic Scenario Search for Vehicular Traffic

Autonomous Vehicles (AVs) are expected to have the potential to impact urban mobility by providing increased safety, reducing traffic congestion, mitigating accidents and reducing emissions. Since AVs operate with little or no human intervention, it is very essential to perceive the external world and understand different objects and their relationships in the scene, and respond appropriately. For doing this effectively, AVs need to be trained on a variety of traffic situations and appropriate responses to them. Behaviour of vehicular traffic varies widely from one part of the world to another. An AV trained for traffic conditions in one part of the world may not be effective, or worse, even be risky in some other part of the world. There is hence a need to create datasets of vehicular traffic scenarios and design mechanisms to query, retrieve and reason about dynamic traffic scenarios. This paper discusses a method for vector based scenario search using a natural language interface for describing traffic scenarios. We first generate textual descriptions of snapshots of traffic scenarios captured from instrumenting vehicles using image captioning libraries. Next, we create vector embeddings of the captions, store them in a vector database to enable semantic scenario search using natural language based queries. This is an ongoing work where different other modalities of scenarios data are planned to be supported over an underlying image captioning and natural language search interface. Experimental results on the image captioning core, show encouraging results.