This week’s post was written by Ray Shen, an undergraduate student from the University of Maryland who is working with the OB team this summer.
Real-Time ArUco Marker Detection in Bonsai-RX
ArUco markers are fiducial markers widely used for camera pose estimation, object tracking, and spatial positioning in computer vision, robotics, and augmented reality. Recently, it was demonstrated that ArUco markers can be used for real-time tracking of rodent engagement during a nose-poking go/no-go task (Smith et al., 2024). Smith et al.’s approach employed Python with the Open Source Computer Vision Library (OpenCV) for marker detection.
Bonsai-RX, a reactive programming language, has gained popularity in behavioral neuroscience due to its intuitive, event-based model and ability to handle multiple asynchronous data streams. It is commonly used for tasks like video acquisition and tracking, closed-loop experiments, and state machines.
Here we describe our effort to integrate Bonsai with ArUco marker detection for real-time behavioral tracking. The camera used was a Logitech C920S HD Pro Webcam, with a frame rate of 30 FPS. The dimensions of the analyzed ArUco marker are 15 mm x 15 mm and the marker was positioned approximately 300 mm from the camera lens. The test was completed with sufficient lighting. We found that the current implementation of the ArUco extension in Bonsai, while functional, has significant limitations for real-time tracking purposes:
- Lack of Rotational Information: Bonsai’s ArUco marker detection system outputs only the marker’s center coordinates and does not provide the positions of the four corners. As a result, real-time rotation or angle tracking is not possible with Bonsai’s ArUco extension (unless multiple markers are incorporated).
- Motion-Related Failures: Real-time detection of ArUco markers often fails during fast movements, likely due to motion blur, as shown in Figure 1. Similar problems were also reported in the study by Smith et al., which did not use Bonsai. While higher frame rate cameras may help in addressing this issue, they pose challenges such as increased costs and large file sizes.
- Other Failures: Detection can also fail under various suboptimal conditions, such as low lighting or poor contrast between the marker and its background, further reducing tracking reliability. Moreover, detection accuracy has been shown to decrease when the marker is positioned too far from the camera (Sampathkrishna, 2022). For this experiment, detection entirely failed at a marker distance of above approximately 350 mm from the camera lens.
Figure 1. ArUco marker trajectory tracked in real-time with Bonsai-RX. Blue: X position of the marker’s center; green: Y position of the marker’s center; red: period of marker detection failure. The marker’s position is indicated in real-world coordinates using homography transformation.
To address these limitations, we developed a Python program for offline (post-hoc) analysis using OpenCV. This program reliably detects and tracks ArUco markers, even in frames where Bonsai failed, and extracts not only the marker’s position coordinates but also its rotation angle over the given time interval, as displayed in Figure 2. This code is available on GitHub: https://github.com/LaubachLab/ArUco-tracker
Figure 2. ArUco marker trajectory successfully extracted from the same video recording used for Figure 1, using offline analysis with Python and the OpenCV library. (a) Blue and green: X and Y positions of the marker’s center, respectively; (b) red: marker angle. The marker’s position is indicated in real-world coordinates using homography transformation.
While Bonsai is a valuable tool in behavioral neuroscience, its current implementation of ArUco markers requires major improvements to enable comprehensive real-time tracking. Enhancements such as providing the corner coordinates of markers and improving robustness to motion blur would significantly extend its functionality. Until these features become available, combining Bonsai’s real-time tracking with OpenCV’s offline analysis offers a practical solution for behavioral applications requiring detailed marker tracking.
References
Sampathkrishna, A. (2022). ArUco Maker based localization and Node graph approach to mapping. arXiv (Cornell University). https://doi.org/10.48550/arxiv.2208.09355
Smith, T. J., Smith, T. R., Faruk, F., Bendea, M., Kumara, S. T., Capadona, J. R., Hernandez-Reynoso, A. G., & Pancrazio, J. J. (n.d.). Real-Time Assessment of Rodent Engagement Using ArUco Markers: A Scalable and Accessible Approach for Scoring Behavior in a Nose-Poking Go/No-Go Task. eNeuro, 11(3), ENEURO.0500-23.2024. https://doi.org/10.1523/eneuro.0500-23.2024