{"id":3811,"date":"2021-03-25T10:29:53","date_gmt":"2021-03-25T09:29:53","guid":{"rendered":"https:\/\/www.ceessnoek.info\/?p=3811"},"modified":"2021-03-25T10:32:09","modified_gmt":"2021-03-25T09:32:09","slug":"cvpr2021-repetitive-activity-counting-by-sight-and-sound","status":"publish","type":"post","link":"https:\/\/www.ceessnoek.info\/index.php\/cvpr2021-repetitive-activity-counting-by-sight-and-sound\/","title":{"rendered":"CVPR 2021: Repetitive Activity Counting by Sight and Sound"},"content":{"rendered":"\n<p>The CVPR 2021 cam-ready &#8220;<em>Repetitive Activity Counting by Sight and Sound<\/em>&#8221; by Yunhua Zhang, Ling Shao and Cees Snoek is <a href=\"https:\/\/arxiv.org\/abs\/2103.13096\">now available<\/a>. This paper strives for repetitive activity counting in videos. Different from existing works, which all analyze the visual video content only, we incorporate for the first time the corresponding sound into the repetition counting process. This benefits accuracy in challenging vision conditions such as occlusion, dramatic camera view changes, low resolution, etc. We propose a model that starts with analyzing the sight and sound streams separately. Then an audiovisual temporal stride decision module and a reliability estimation module are introduced to exploit cross-modal temporal interaction. For learning and evaluation, an existing dataset is repurposed and reorganized to allow for repetition counting with sight and sound. We also introduce a variant of this dataset for repetition counting under challenging vision conditions. Experiments demonstrate the benefit of sound, as well as the other introduced modules, for repetition counting. Our sight-only model already outperforms the state-of-the-art by itself, when we add sound, results improve notably, especially under harsh vision conditions. The code and datasets are available at <a href=\"https:\/\/github.com\/xiaobai1217\/RepetitionCounting\">https:\/\/github.com\/xiaobai1217\/RepetitionCounting<\/a>.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><a href=\"https:\/\/www.ceessnoek.info\/wp-content\/uploads\/2021\/03\/repetition-sight-sound.png\"><img loading=\"lazy\" decoding=\"async\" width=\"936\" height=\"948\" src=\"https:\/\/www.ceessnoek.info\/wp-content\/uploads\/2021\/03\/repetition-sight-sound.png\" alt=\"\" class=\"wp-image-3798\" srcset=\"https:\/\/www.ceessnoek.info\/wp-content\/uploads\/2021\/03\/repetition-sight-sound.png 936w, https:\/\/www.ceessnoek.info\/wp-content\/uploads\/2021\/03\/repetition-sight-sound-296x300.png 296w, https:\/\/www.ceessnoek.info\/wp-content\/uploads\/2021\/03\/repetition-sight-sound-768x778.png 768w\" sizes=\"auto, (max-width: 936px) 100vw, 936px\" \/><\/a><\/figure>\n","protected":false},"excerpt":{"rendered":"<p>The CVPR 2021 cam-ready &#8220;Repetitive Activity Counting by Sight and Sound&#8221; by Yunhua Zhang, Ling Shao and Cees Snoek is now available. This paper strives for repetitive activity counting in videos. Different from existing works, which all analyze the visual video content only, we incorporate for the first time the corresponding sound into the repetition [&hellip;]<\/p>\n","protected":false},"author":3,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[4],"tags":[],"class_list":["post-3811","post","type-post","status-publish","format-standard","hentry","category-science"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v25.3 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>CVPR 2021: Repetitive Activity Counting by Sight and Sound - Cees Snoek<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.ceessnoek.info\/index.php\/cvpr2021-repetitive-activity-counting-by-sight-and-sound\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"CVPR 2021: Repetitive Activity Counting by Sight and Sound - Cees Snoek\" \/>\n<meta property=\"og:description\" content=\"The CVPR 2021 cam-ready &#8220;Repetitive Activity Counting by Sight and Sound&#8221; by Yunhua Zhang, Ling Shao and Cees Snoek is now available. This paper strives for repetitive activity counting in videos. Different from existing works, which all analyze the visual video content only, we incorporate for the first time the corresponding sound into the repetition [&hellip;]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.ceessnoek.info\/index.php\/cvpr2021-repetitive-activity-counting-by-sight-and-sound\/\" \/>\n<meta property=\"og:site_name\" content=\"Cees Snoek\" \/>\n<meta property=\"article:published_time\" content=\"2021-03-25T09:29:53+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2021-03-25T09:32:09+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.ceessnoek.info\/wp-content\/uploads\/2021\/03\/repetition-sight-sound.png\" \/>\n<meta name=\"author\" content=\"Cees\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Cees\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"2 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.ceessnoek.info\/index.php\/cvpr2021-repetitive-activity-counting-by-sight-and-sound\/\",\"url\":\"https:\/\/www.ceessnoek.info\/index.php\/cvpr2021-repetitive-activity-counting-by-sight-and-sound\/\",\"name\":\"CVPR 2021: Repetitive Activity Counting by Sight and Sound - Cees Snoek\",\"isPartOf\":{\"@id\":\"https:\/\/www.ceessnoek.info\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/www.ceessnoek.info\/index.php\/cvpr2021-repetitive-activity-counting-by-sight-and-sound\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/www.ceessnoek.info\/index.php\/cvpr2021-repetitive-activity-counting-by-sight-and-sound\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.ceessnoek.info\/wp-content\/uploads\/2021\/03\/repetition-sight-sound.png\",\"datePublished\":\"2021-03-25T09:29:53+00:00\",\"dateModified\":\"2021-03-25T09:32:09+00:00\",\"author\":{\"@id\":\"https:\/\/www.ceessnoek.info\/#\/schema\/person\/4bca975b7c432aeb5dced40bdbc204c1\"},\"breadcrumb\":{\"@id\":\"https:\/\/www.ceessnoek.info\/index.php\/cvpr2021-repetitive-activity-counting-by-sight-and-sound\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.ceessnoek.info\/index.php\/cvpr2021-repetitive-activity-counting-by-sight-and-sound\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.ceessnoek.info\/index.php\/cvpr2021-repetitive-activity-counting-by-sight-and-sound\/#primaryimage\",\"url\":\"https:\/\/www.ceessnoek.info\/wp-content\/uploads\/2021\/03\/repetition-sight-sound.png\",\"contentUrl\":\"https:\/\/www.ceessnoek.info\/wp-content\/uploads\/2021\/03\/repetition-sight-sound.png\",\"width\":936,\"height\":948},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.ceessnoek.info\/index.php\/cvpr2021-repetitive-activity-counting-by-sight-and-sound\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.ceessnoek.info\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"CVPR 2021: Repetitive Activity Counting by Sight and Sound\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.ceessnoek.info\/#website\",\"url\":\"https:\/\/www.ceessnoek.info\/\",\"name\":\"Cees Snoek\",\"description\":\"research on video and image ai\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.ceessnoek.info\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.ceessnoek.info\/#\/schema\/person\/4bca975b7c432aeb5dced40bdbc204c1\",\"name\":\"Cees\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.ceessnoek.info\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/756ccb993852c1e8e3af39a228d11a7305b2a937750f26dc5799d5df019b0f51?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/756ccb993852c1e8e3af39a228d11a7305b2a937750f26dc5799d5df019b0f51?s=96&d=mm&r=g\",\"caption\":\"Cees\"},\"sameAs\":[\"http:\/\/www.CeesSnoek.info\"],\"url\":\"https:\/\/www.ceessnoek.info\/index.php\/author\/admin\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"CVPR 2021: Repetitive Activity Counting by Sight and Sound - Cees Snoek","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.ceessnoek.info\/index.php\/cvpr2021-repetitive-activity-counting-by-sight-and-sound\/","og_locale":"en_US","og_type":"article","og_title":"CVPR 2021: Repetitive Activity Counting by Sight and Sound - Cees Snoek","og_description":"The CVPR 2021 cam-ready &#8220;Repetitive Activity Counting by Sight and Sound&#8221; by Yunhua Zhang, Ling Shao and Cees Snoek is now available. This paper strives for repetitive activity counting in videos. Different from existing works, which all analyze the visual video content only, we incorporate for the first time the corresponding sound into the repetition [&hellip;]","og_url":"https:\/\/www.ceessnoek.info\/index.php\/cvpr2021-repetitive-activity-counting-by-sight-and-sound\/","og_site_name":"Cees Snoek","article_published_time":"2021-03-25T09:29:53+00:00","article_modified_time":"2021-03-25T09:32:09+00:00","og_image":[{"url":"https:\/\/www.ceessnoek.info\/wp-content\/uploads\/2021\/03\/repetition-sight-sound.png","type":"","width":"","height":""}],"author":"Cees","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Cees","Est. reading time":"2 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/www.ceessnoek.info\/index.php\/cvpr2021-repetitive-activity-counting-by-sight-and-sound\/","url":"https:\/\/www.ceessnoek.info\/index.php\/cvpr2021-repetitive-activity-counting-by-sight-and-sound\/","name":"CVPR 2021: Repetitive Activity Counting by Sight and Sound - Cees Snoek","isPartOf":{"@id":"https:\/\/www.ceessnoek.info\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.ceessnoek.info\/index.php\/cvpr2021-repetitive-activity-counting-by-sight-and-sound\/#primaryimage"},"image":{"@id":"https:\/\/www.ceessnoek.info\/index.php\/cvpr2021-repetitive-activity-counting-by-sight-and-sound\/#primaryimage"},"thumbnailUrl":"https:\/\/www.ceessnoek.info\/wp-content\/uploads\/2021\/03\/repetition-sight-sound.png","datePublished":"2021-03-25T09:29:53+00:00","dateModified":"2021-03-25T09:32:09+00:00","author":{"@id":"https:\/\/www.ceessnoek.info\/#\/schema\/person\/4bca975b7c432aeb5dced40bdbc204c1"},"breadcrumb":{"@id":"https:\/\/www.ceessnoek.info\/index.php\/cvpr2021-repetitive-activity-counting-by-sight-and-sound\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.ceessnoek.info\/index.php\/cvpr2021-repetitive-activity-counting-by-sight-and-sound\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.ceessnoek.info\/index.php\/cvpr2021-repetitive-activity-counting-by-sight-and-sound\/#primaryimage","url":"https:\/\/www.ceessnoek.info\/wp-content\/uploads\/2021\/03\/repetition-sight-sound.png","contentUrl":"https:\/\/www.ceessnoek.info\/wp-content\/uploads\/2021\/03\/repetition-sight-sound.png","width":936,"height":948},{"@type":"BreadcrumbList","@id":"https:\/\/www.ceessnoek.info\/index.php\/cvpr2021-repetitive-activity-counting-by-sight-and-sound\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.ceessnoek.info\/"},{"@type":"ListItem","position":2,"name":"CVPR 2021: Repetitive Activity Counting by Sight and Sound"}]},{"@type":"WebSite","@id":"https:\/\/www.ceessnoek.info\/#website","url":"https:\/\/www.ceessnoek.info\/","name":"Cees Snoek","description":"research on video and image ai","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.ceessnoek.info\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/www.ceessnoek.info\/#\/schema\/person\/4bca975b7c432aeb5dced40bdbc204c1","name":"Cees","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.ceessnoek.info\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/756ccb993852c1e8e3af39a228d11a7305b2a937750f26dc5799d5df019b0f51?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/756ccb993852c1e8e3af39a228d11a7305b2a937750f26dc5799d5df019b0f51?s=96&d=mm&r=g","caption":"Cees"},"sameAs":["http:\/\/www.CeesSnoek.info"],"url":"https:\/\/www.ceessnoek.info\/index.php\/author\/admin\/"}]}},"_links":{"self":[{"href":"https:\/\/www.ceessnoek.info\/index.php\/wp-json\/wp\/v2\/posts\/3811","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.ceessnoek.info\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.ceessnoek.info\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.ceessnoek.info\/index.php\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/www.ceessnoek.info\/index.php\/wp-json\/wp\/v2\/comments?post=3811"}],"version-history":[{"count":3,"href":"https:\/\/www.ceessnoek.info\/index.php\/wp-json\/wp\/v2\/posts\/3811\/revisions"}],"predecessor-version":[{"id":3815,"href":"https:\/\/www.ceessnoek.info\/index.php\/wp-json\/wp\/v2\/posts\/3811\/revisions\/3815"}],"wp:attachment":[{"href":"https:\/\/www.ceessnoek.info\/index.php\/wp-json\/wp\/v2\/media?parent=3811"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.ceessnoek.info\/index.php\/wp-json\/wp\/v2\/categories?post=3811"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.ceessnoek.info\/index.php\/wp-json\/wp\/v2\/tags?post=3811"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}