Cees Snoek

Google (and YouTube for that matter)  searches video using closed captions, user-provided tags, or text embedded in web pages. Based on visual evidence obtained from Google tech talks by Alex Hauptmann and John Smith I have reason to believe that concept-based video retrieval, i.e. searching based on the visual content, is receiving more and more attention at Google. The video frame on the left shows the audience at the TechTalk by Alex on March 1st 2006, while the frame on the right shows the audience at John’s talk on October 25, 2007. Note the audience increase. I wonder how many people will attend the next Google TechTalk on this topic ;)

Concept-based video search receives more an more attention at Google. On the left the number of attendees during a talk on the topic in early 2006, on the right the number of attendees during a talk in late 2007.

The paper Balancing Thread Based Navigation for Targeted Video Search by Ork de Rooij, Cees Snoek, and Marcel Worring is available online now. Various query methods for video search exist. Because of the semantic gap each method has its limitations. We argue that for effective retrieval query methods need to be combined at retrieval time. However, switching query methods often involves a change in query and browsing interface, which puts a heavy burden on the user. In this paper, we propose a novel method for fast and effective search through large video collections by embedding multiple query methods into a single browsing environment. To that end we introduced the notion of query threads, which contain a shot-based ranking of the video collection according to some feature-based similarity measure. On top of these threads we define several thread-based visualizations, ranging from fast targeted search to very broad exploratory search, with the ForkBrowser as the balance between fast search and video space exploration. We compare the effectiveness and efficiency of the ForkBrowser with the CrossBrowser on the TRECVID 2007 interactive search task. Results show that different query methods are needed for different types of search topics, and that the ForkBrowser requires signifficantly less user interactions to achieve the same result as the CrossBrowser. In addition, both browsers rank among the best interactive retrieval systems currently available.

The paper A Comparison of Color Features for Visual Concept Classification by Koen van de Sande, Theo Gevers, and Cees Snoek is available online now. The paper improves upon the CVPR 2008 paper below by considering also the points at which color features are extracted. Hence, different point sampling strategies based on Harris-Laplace salient points, dense sampling and the spatial pyramid are studied in concert with color features.

The paper Evaluation of Color Descriptors for Object and Scene Recognition by Koen van de Sande, Theo Gevers, and Cees Snoek is available online now. Image category recognition is important to access visual information on the level of objects and scene types. So far, intensity-based descriptors have been widely used. To increase illumination invariance and discriminative power, color descriptors have been proposed only recently. As many descriptors exist, a structured overview of color invariant descriptors in the context of image category recognition is required. Therefore, this paper studies the invariance properties and the distinctiveness of color descriptors in a structured way. The invariance properties of color descriptors are shown analytically using a taxonomy based on invariance properties with respect to photometric transformations. The distinctiveness of color descriptors is assessed experimentally using two benchmarks from the image domain and the video domain. From the theoretical and experimental results, it can be derived that invariance to light intensity changes and light color changes affects category recognition. The results reveal further that, for light intensity changes, the usefulness of invariance is category-specific.

During the past two weeks I spent some pleasant time in Germany for a Dagstuhl meeting, the ACM Multimedia PC meeting, and the ICME conference. My personal highlight of all events was the Dagstuhl meeting on Contextual and Social Media Understanding and Usage. This was a very interesting experience. The Dagstuhl concept works as follows: you lock a bunch of people that work on a similar research topic in a castle-like venue in the middle of nowhere. You make sure that they are fed at regular intervals during the day, you group them based on specific interests, and what will happen is that they start to think and talk about novel research ideas. I ended up in a group with Alex Hauptmann, Gareth Jones, and Stéphane Marchand-Maillet. We discussed the possibility of opening a huge central repository for multimedia data, features, software, and experimental results to boost research in contextual and social media understanding. I hope this idea will generate some activity the coming months. Apart from discussing the social aspects of multimedia research, the Dagstuhl meeting was also a very engaging social activity, I especially liked the discussions in the wine cellar :) I owe a big thanks to the organizers of the workshop: Susanne Boll, Mohan Kankanhalli, Gopal Pingali, and Svetha Venkatesh, for inviting me, and of course to the fellow participants (see group photo) for the great atmosphere. I hope to come back to this place at some later point in time.

The BeeldCanon project proposal by Luc Van Gool, Tinne Tuytelaars, Arnold Smeulders and Cees Snoek has been granted a subsidy by the Dutch ICTRegie and Flemish IBBT. The project strives to make Dutch/Flemish image culture, as captured in the video archives of Beeld en Geluid and VRT, accessible to the public. To that end we aim to detect the most important Dutch/Flemish monuments and sceneries, like the Atomium and the Zaanse Schans, and make this (extendible) set of visual detectors available for general-purpose visual retrieval. The scientific challenge lies in realizing a solution which is both highly efficient and highly effective. The project builds on KU Leuven’s expertise in invariant visual representations and object recognition, and the expertise of the University of Amsterdam in large-scale video analysis and interactive retrieval. Prospective PhD students who are interested in the topic are welcome to contact me for more details.

VideOlympics: Real-Time Evaluation of Multimedia Retrieval Systems by Cees Snoek, Marcel Worring, Ork de Rooij, Koen van de Sande, Rong Yan, and Alex Hauptmann has now been published in IEEE MultiMedia. Demo sessions of multimedia retrieval systems are ideal venues to disseminate scientific results. Existing demo sessions, however, fail to engage the audience fully. We argue that real-time evaluation of several multimedia retrieval systems in a single showcase increases impact. We highlight the requirements for such a focused effort and share our experience in organizing the VideOlympics: a real-time evaluation showcase of present day video search engines. We produced a video trailer of the event (in collaboration with Fabchannel), which gives a multimedia impression of the first VideOlympics as recorded on July 10, 2007 at Beeld en Geluid, the Dutch national broadcast archive.

We organize another VideOlympics at CIVR 2008.

A recent study reveals a worldwide downtrend in visits to national parks. The authors show that typical decline started between 1981 and 1991, and that this lack of interest is potentially hurting current conservation efforts. According to the authors a possible cause is what they call videophilia:

“The new human tendency to focus on sedentary activities involving electronic media.”

So in effect, research aiming for better video search methodology is hurting biodiversity conservation. Pfff.