Following several conversations on Twitter about methods for video analysis, in May 2021 the OpenBehavior team reached out to prominent research groups working on methods for video analysis and groups who contributed videos to our behavioral neuroscience video repository to organize what we called Community Conversations on Video Analysis. Two sessions have been held over the past two weeks via Zoom, with up to 30 participants, and we are planning to continue the conversations over this summer.

The first session focused on Sharing & Benchmarks. Mark Laubach and Greg Corder led a discussion on why sharing video data, especially raw videos, is very much needed to advance the field (training, education, rigor and reproducibility). Talmo Pereira then discussed data formats for video recordings and proposed his own approach to a new structure for video data and meta data (e.g. landmarks, detection scores). Ann Kennedy then presented her work on the Multi-Agent Behavior (MABe) challenge, which has a goal of automating the annotation process that underlies video methods such as pose estimation.

The second session focused on machine learning tools for video analysis. The goal was to attempt to answer a question posted by Alik Widge on Twitter on May 5: “Has anyone yet shown whether the unsupervised clustering of pose estimation produces similar clusters/motifs across labs?” We spent about 20 minutes discussing this question, and the general consensus was the answer to this question is no, and that having good benchmark data sets might make it possible to at least answer the question positively across analysis methods. Three current machine learning approaches were then discussed: VAME by Kevin Luxem, B-SOiD by Alex Hsu, and TREBA developed by Jennifer Sun and the Anderson and Perona labs at CalTech, and presented by Ann Kennedy along with some other tools.

We decided to meet regularly on Mondays at noon ET. Our next session will include a presentation by Sam Golden on SIMBA and a discussion of SHAP, a recent and popular approach for model interpretation in machine learning. (This blog post is a nice summary of the method.) The rest of the session will cover topics that have been raised in a Slack channel maintained by the OpenBehavior team and including participants from the Zoom sessions.

If you are a video analysis developer, a power user of an existing method, a fan of these methods, or just curious and wish to participate, please get in touch by sending an email to the OpenBehavior team using this LINK.