A Large-Scale Visual Check-In System for TV Content-Aware Web with Client-Side Video Analysis Offloading

The intuitive linkage between TV and the web brings about new opportunities to motivate people to watch video content or visit websites. A check-in system that recognizes which specific programs are being watched by users is highly effective in promoting TV content. However, such a check-in system faces two technical problems: the temporal characteristics of broadcasting media, resulting in a massive number of simultaneous check-in requests, and the wide variation of audience environments, such as lighting, cameras, and TV devices. We propose a visual check-in system for linking websites and TV programs. The system identifies what program a user is watching by analyzing the visual features of a video captured with a smartphone. The key technology is a real-time video analysis framework that achieves both scalability to an enormous number of simultaneous requests and practical robustness in terms of content identification. We have constructed a special color scheme consisting of 120 (non-neutral) colors to absorb differences in the illumination levels of user environments. This color scheme plays an important role in offloading video analysis tasks onto the client-side in a tamper-proof way. Our system assigns a unique color scheme to each user and verifies a check-in request using the corresponding color scheme, thus preventing malicious users from sharing the analysis results with others. Experimental results using a real dataset demonstrate the accuracy and efficiency of the proposed method. We have applied the system to actual TV programs and clarified its scalability and precision in a production environment.