Extreme event detection and assimilation from multimedia sources

A new event-based multimedia processing framework for detection, retrieval, and cross-media content assimilation of geo-spatiotemporal phenomena is described. Multimedia information relevant to geo-spatiotemporal events are available from sources such as remote satellites, in-situ sensors as image streams, and other outlets such as news articles, weather bulletins as text documents and in various formats, each with widely varying properties. We pose an event-based framework to automatically detect geo-spatiotemporal phenomena from raw untagged remote-sensing satellite image streams, extract attributes for such events, match the spatiotemporal properties of the detected phenomenon with events in a database to automatically derive a media-independent event description, and subsequently use the media-independent event descriptions to assimilate relevant information of the same event across other media sources from the web using a mashup. A virtual globe interface enables simultaneous visualization of the assimilated spatiotemporal information annotated with Internet sources. This framework is demonstrated for the automatic detection of tropical cyclones from satellite imagery followed by the retrieval and assimilation of related information from government-run weather sites and commercial news portals.