Cees Snoek

Our TRECVID 2008 workshop paper entitled “The MediaMill TRECVID 2008 Semantic Video Search Engine” is available online. In this paper we describe our TRECVID 2008 video retrieval experiments. The MediaMill team participated in three tasks: concept detection, automatic search, and interactive search. Rather than continuing to increase the number of concept detectors available for retrieval, our TRECVID 2008 experiments focus on increasing the robustness of a small set of detectors. To that end, our concept detection experiments emphasize in particular the role of sampling, the value of color invariant features, the influence of codebook construction, and the effectiveness of kernel-based learning parameters. For retrieval, a robust but limited set of concept detectors necessitates the need to rely on as many auxiliary information channels as possible. Therefore, our automatic search experiments focus on predicting which information channel to trust given a certain topic, leading to a novel framework for predictive video retrieval. To improve the video retrieval results further, our interactive search experiments investigate the roles of visualizing preview results for a certain browse-dimension and active learning mechanisms that learn to solve complex search topics by analysis from user browsing behavior. The 2008 edition of the TRECVID benchmark has been the most successful MediaMill participation to date, resulting in the top ranking for both concept detection and interactive search, and a runner-up ranking for automatic retrieval. Again a lot has been learned during this year’s TRECVID campaign; we highlight the most important lessons at the end of this paper.

The University of Amsterdam is on the lookout for a professor specializing in historic Dutch multimedia, or someone who studies the Dutch media developments in radio, television, and quite likely the Internet, over the past century. The vacancy is supported by the Netherlands Institute for Sound and Vision, not accidentally a close collaborator of our group also. I wonder if our semantic video search engine would be of any use for the new professor and her/his team. To be continued?

Google (and YouTube for that matter)  searches video using closed captions, user-provided tags, or text embedded in web pages. Based on visual evidence obtained from Google tech talks by Alex Hauptmann and John Smith I have reason to believe that concept-based video retrieval, i.e. searching based on the visual content, is receiving more and more attention at Google. The video frame on the left shows the audience at the TechTalk by Alex on March 1st 2006, while the frame on the right shows the audience at John’s talk on October 25, 2007. Note the audience increase. I wonder how many people will attend the next Google TechTalk on this topic ;)

Concept-based video search receives more an more attention at Google. On the left the number of attendees during a talk on the topic in early 2006, on the right the number of attendees during a talk in late 2007.

The paper Balancing Thread Based Navigation for Targeted Video Search by Ork de Rooij, Cees Snoek, and Marcel Worring is available online now. Various query methods for video search exist. Because of the semantic gap each method has its limitations. We argue that for effective retrieval query methods need to be combined at retrieval time. However, switching query methods often involves a change in query and browsing interface, which puts a heavy burden on the user. In this paper, we propose a novel method for fast and effective search through large video collections by embedding multiple query methods into a single browsing environment. To that end we introduced the notion of query threads, which contain a shot-based ranking of the video collection according to some feature-based similarity measure. On top of these threads we define several thread-based visualizations, ranging from fast targeted search to very broad exploratory search, with the ForkBrowser as the balance between fast search and video space exploration. We compare the effectiveness and efficiency of the ForkBrowser with the CrossBrowser on the TRECVID 2007 interactive search task. Results show that different query methods are needed for different types of search topics, and that the ForkBrowser requires signifficantly less user interactions to achieve the same result as the CrossBrowser. In addition, both browsers rank among the best interactive retrieval systems currently available.