{"id":1840,"date":"2019-03-09T12:40:06","date_gmt":"2019-03-09T11:40:06","guid":{"rendered":"http:\/\/www.ceessnoek.info\/?p=1840"},"modified":"2019-03-09T14:44:33","modified_gmt":"2019-03-09T13:44:33","slug":"pointly-supervised-action-localization","status":"publish","type":"post","link":"https:\/\/www.ceessnoek.info\/index.php\/pointly-supervised-action-localization\/","title":{"rendered":"Pointly-Supervised Action Localization"},"content":{"rendered":"<p><img loading=\"lazy\" decoding=\"async\" class=\"alignright size-medium wp-image-1843\" src=\"http:\/\/www.ceessnoek.info\/wp-content\/uploads\/2019\/03\/Screenshot-2019-03-09-at-12.39.10-300x222.png\" alt=\"\" width=\"300\" height=\"222\" srcset=\"https:\/\/www.ceessnoek.info\/wp-content\/uploads\/2019\/03\/Screenshot-2019-03-09-at-12.39.10-300x222.png 300w, https:\/\/www.ceessnoek.info\/wp-content\/uploads\/2019\/03\/Screenshot-2019-03-09-at-12.39.10-768x568.png 768w, https:\/\/www.ceessnoek.info\/wp-content\/uploads\/2019\/03\/Screenshot-2019-03-09-at-12.39.10-1024x757.png 1024w, https:\/\/www.ceessnoek.info\/wp-content\/uploads\/2019\/03\/Screenshot-2019-03-09-at-12.39.10.png 1372w\" sizes=\"auto, (max-width: 300px) 100vw, 300px\" \/>The paper &#8220;Pointly-Supervised Action Localization&#8221; by Pascal Mettes and Cees Snoek has been published in the International Journal of Computer Vision. <a href=\"https:\/\/doi.org\/10.1007\/s11263-018-1120-4\">The paper<\/a> strives for spatio-temporal localization of human actions in videos. In the literature, the consensus is to achieve localization by training on bounding box annotations provided for each frame of each training video. As annotating boxes in video is expensive, cumbersome and error-prone, we propose to bypass box-supervision. Instead, we introduce action localization based on point-supervision. We start from unsupervised spatio-temporal proposals, which provide a set of candidate regions in videos. While normally used exclusively for inference, we show spatio-temporal proposals can also be leveraged during training when guided by a sparse set of point annotations. We introduce an overlap measure between points and spatio-temporal proposals and incorporate them all into a new objective of a Multiple Instance Learning optimization. During inference, we introduce pseudo-points, visual cues from videos, that automatically guide the selection of spatio-temporal proposals. We outline five spatial and one temporal pseudo-point, as well as a measure to best leverage pseudo-points at test time. Experimental evaluation on three action localization datasets shows our pointly-supervised approach (i) is as effective as traditional box-supervision at a fraction of the annotation cost, (ii) is robust to sparse and noisy point annotations, (iii) benefits from pseudo-points during inference, and (iv) outperforms recent weakly-supervised alternatives. This leads us to conclude that points provide a viable alternative to boxes for action localization.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>The paper &#8220;Pointly-Supervised Action Localization&#8221; by Pascal Mettes and Cees Snoek has been published in the International Journal of Computer Vision. The paper strives for spatio-temporal localization of human actions in videos. In the literature, the consensus is to achieve localization by training on bounding box annotations provided for each frame of each training video. [&hellip;]<\/p>\n","protected":false},"author":3,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-1840","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v25.3 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Pointly-Supervised Action Localization - Cees Snoek<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.ceessnoek.info\/index.php\/pointly-supervised-action-localization\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Pointly-Supervised Action Localization - Cees Snoek\" \/>\n<meta property=\"og:description\" content=\"The paper &#8220;Pointly-Supervised Action Localization&#8221; by Pascal Mettes and Cees Snoek has been published in the International Journal of Computer Vision. The paper strives for spatio-temporal localization of human actions in videos. In the literature, the consensus is to achieve localization by training on bounding box annotations provided for each frame of each training video. [&hellip;]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.ceessnoek.info\/index.php\/pointly-supervised-action-localization\/\" \/>\n<meta property=\"og:site_name\" content=\"Cees Snoek\" \/>\n<meta property=\"article:published_time\" content=\"2019-03-09T11:40:06+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2019-03-09T13:44:33+00:00\" \/>\n<meta property=\"og:image\" content=\"http:\/\/www.ceessnoek.info\/wp-content\/uploads\/2019\/03\/Screenshot-2019-03-09-at-12.39.10-300x222.png\" \/>\n<meta name=\"author\" content=\"Cees\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Cees\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"1 minute\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.ceessnoek.info\/index.php\/pointly-supervised-action-localization\/\",\"url\":\"https:\/\/www.ceessnoek.info\/index.php\/pointly-supervised-action-localization\/\",\"name\":\"Pointly-Supervised Action Localization - Cees Snoek\",\"isPartOf\":{\"@id\":\"https:\/\/www.ceessnoek.info\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/www.ceessnoek.info\/index.php\/pointly-supervised-action-localization\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/www.ceessnoek.info\/index.php\/pointly-supervised-action-localization\/#primaryimage\"},\"thumbnailUrl\":\"http:\/\/www.ceessnoek.info\/wp-content\/uploads\/2019\/03\/Screenshot-2019-03-09-at-12.39.10-300x222.png\",\"datePublished\":\"2019-03-09T11:40:06+00:00\",\"dateModified\":\"2019-03-09T13:44:33+00:00\",\"author\":{\"@id\":\"https:\/\/www.ceessnoek.info\/#\/schema\/person\/4bca975b7c432aeb5dced40bdbc204c1\"},\"breadcrumb\":{\"@id\":\"https:\/\/www.ceessnoek.info\/index.php\/pointly-supervised-action-localization\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.ceessnoek.info\/index.php\/pointly-supervised-action-localization\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.ceessnoek.info\/index.php\/pointly-supervised-action-localization\/#primaryimage\",\"url\":\"https:\/\/www.ceessnoek.info\/wp-content\/uploads\/2019\/03\/Screenshot-2019-03-09-at-12.39.10.png\",\"contentUrl\":\"https:\/\/www.ceessnoek.info\/wp-content\/uploads\/2019\/03\/Screenshot-2019-03-09-at-12.39.10.png\",\"width\":1372,\"height\":1014},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.ceessnoek.info\/index.php\/pointly-supervised-action-localization\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.ceessnoek.info\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Pointly-Supervised Action Localization\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.ceessnoek.info\/#website\",\"url\":\"https:\/\/www.ceessnoek.info\/\",\"name\":\"Cees Snoek\",\"description\":\"research on video and image ai\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.ceessnoek.info\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.ceessnoek.info\/#\/schema\/person\/4bca975b7c432aeb5dced40bdbc204c1\",\"name\":\"Cees\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.ceessnoek.info\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/756ccb993852c1e8e3af39a228d11a7305b2a937750f26dc5799d5df019b0f51?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/756ccb993852c1e8e3af39a228d11a7305b2a937750f26dc5799d5df019b0f51?s=96&d=mm&r=g\",\"caption\":\"Cees\"},\"sameAs\":[\"http:\/\/www.CeesSnoek.info\"],\"url\":\"https:\/\/www.ceessnoek.info\/index.php\/author\/admin\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Pointly-Supervised Action Localization - Cees Snoek","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.ceessnoek.info\/index.php\/pointly-supervised-action-localization\/","og_locale":"en_US","og_type":"article","og_title":"Pointly-Supervised Action Localization - Cees Snoek","og_description":"The paper &#8220;Pointly-Supervised Action Localization&#8221; by Pascal Mettes and Cees Snoek has been published in the International Journal of Computer Vision. The paper strives for spatio-temporal localization of human actions in videos. In the literature, the consensus is to achieve localization by training on bounding box annotations provided for each frame of each training video. [&hellip;]","og_url":"https:\/\/www.ceessnoek.info\/index.php\/pointly-supervised-action-localization\/","og_site_name":"Cees Snoek","article_published_time":"2019-03-09T11:40:06+00:00","article_modified_time":"2019-03-09T13:44:33+00:00","og_image":[{"url":"http:\/\/www.ceessnoek.info\/wp-content\/uploads\/2019\/03\/Screenshot-2019-03-09-at-12.39.10-300x222.png","type":"","width":"","height":""}],"author":"Cees","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Cees","Est. reading time":"1 minute"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/www.ceessnoek.info\/index.php\/pointly-supervised-action-localization\/","url":"https:\/\/www.ceessnoek.info\/index.php\/pointly-supervised-action-localization\/","name":"Pointly-Supervised Action Localization - Cees Snoek","isPartOf":{"@id":"https:\/\/www.ceessnoek.info\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.ceessnoek.info\/index.php\/pointly-supervised-action-localization\/#primaryimage"},"image":{"@id":"https:\/\/www.ceessnoek.info\/index.php\/pointly-supervised-action-localization\/#primaryimage"},"thumbnailUrl":"http:\/\/www.ceessnoek.info\/wp-content\/uploads\/2019\/03\/Screenshot-2019-03-09-at-12.39.10-300x222.png","datePublished":"2019-03-09T11:40:06+00:00","dateModified":"2019-03-09T13:44:33+00:00","author":{"@id":"https:\/\/www.ceessnoek.info\/#\/schema\/person\/4bca975b7c432aeb5dced40bdbc204c1"},"breadcrumb":{"@id":"https:\/\/www.ceessnoek.info\/index.php\/pointly-supervised-action-localization\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.ceessnoek.info\/index.php\/pointly-supervised-action-localization\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.ceessnoek.info\/index.php\/pointly-supervised-action-localization\/#primaryimage","url":"https:\/\/www.ceessnoek.info\/wp-content\/uploads\/2019\/03\/Screenshot-2019-03-09-at-12.39.10.png","contentUrl":"https:\/\/www.ceessnoek.info\/wp-content\/uploads\/2019\/03\/Screenshot-2019-03-09-at-12.39.10.png","width":1372,"height":1014},{"@type":"BreadcrumbList","@id":"https:\/\/www.ceessnoek.info\/index.php\/pointly-supervised-action-localization\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.ceessnoek.info\/"},{"@type":"ListItem","position":2,"name":"Pointly-Supervised Action Localization"}]},{"@type":"WebSite","@id":"https:\/\/www.ceessnoek.info\/#website","url":"https:\/\/www.ceessnoek.info\/","name":"Cees Snoek","description":"research on video and image ai","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.ceessnoek.info\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/www.ceessnoek.info\/#\/schema\/person\/4bca975b7c432aeb5dced40bdbc204c1","name":"Cees","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.ceessnoek.info\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/756ccb993852c1e8e3af39a228d11a7305b2a937750f26dc5799d5df019b0f51?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/756ccb993852c1e8e3af39a228d11a7305b2a937750f26dc5799d5df019b0f51?s=96&d=mm&r=g","caption":"Cees"},"sameAs":["http:\/\/www.CeesSnoek.info"],"url":"https:\/\/www.ceessnoek.info\/index.php\/author\/admin\/"}]}},"_links":{"self":[{"href":"https:\/\/www.ceessnoek.info\/index.php\/wp-json\/wp\/v2\/posts\/1840","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.ceessnoek.info\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.ceessnoek.info\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.ceessnoek.info\/index.php\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/www.ceessnoek.info\/index.php\/wp-json\/wp\/v2\/comments?post=1840"}],"version-history":[{"count":2,"href":"https:\/\/www.ceessnoek.info\/index.php\/wp-json\/wp\/v2\/posts\/1840\/revisions"}],"predecessor-version":[{"id":1845,"href":"https:\/\/www.ceessnoek.info\/index.php\/wp-json\/wp\/v2\/posts\/1840\/revisions\/1845"}],"wp:attachment":[{"href":"https:\/\/www.ceessnoek.info\/index.php\/wp-json\/wp\/v2\/media?parent=1840"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.ceessnoek.info\/index.php\/wp-json\/wp\/v2\/categories?post=1840"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.ceessnoek.info\/index.php\/wp-json\/wp\/v2\/tags?post=1840"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}