WACV 2027 · workshop version at CV4Ecology, ECCV 2026

Automated Temporal Localization of Behavior in Cichlid Fish with Few‑Shot Trajectory Tokens

Bhargav Srinivasan1,2,*, William Lamousin1,*, Jason Liu1,†, Charles Phan1,†, Ofure Osunbor2, Shubh Sharma1, Cade McGeehan2, Tahir Haroon2, Vijay Jayasuriya1, Coltan Parker2, Scott Juntti2, Pulkit Kumar1, Abhinav Shrivastava1

University of Maryland, College Park · 1Dept. of Computer Science · 2Dept. of Biology · * † Equal contribution

Paper Code Data Models

Research Problem

Behavioral experiments are limited by how fast experts can watch video.

  • Experts annotate at real-time playback, hour by hour, in BORIS.
  • Four observers for the dataset, reconciled by inter-rater reliability.
  • Each label is a single time point. No start or end timestamps.
17
hour-long recordings
8,202
expert point annotations
67.7%
fall within 2 s of another annotation
Raw recording · 25 fps
Expert point annotations a timestamp and a class 0 s 60 s FSH-TAL output start, end, and class, including overlaps Bite Chase Lead Peck Quiver Tilt

Dataset

Six short, visually similar, often overlapping behaviors.

17 hour-long recordings of Astatotilapia burtoni pairs, side view, 25 fps.

Bite3,516 events

Chase846 events

Lead1,455 events

Peck529 events

Quiver1,262 events

Tilt594 events

Tracking

Segmentation based keypoint tracking.

SAM3 segments both fish in every frame, and points are tracked across the masks to give each fish a trajectory.

No skeleton or keypoint labels provided.

Raw clip

SAM3 segmentation

Point tracking

Lead
Chase

Model

A few-shot classifier feeds a point-supervised localizer.

Few-shot MIL classifier
Trajectory-aligned tokens at the tracked points fused with motion features, trained in few-shot episodes.
Point-supervised localizer
Behavior, timespan, and centerness heads, trained on ±2 s pseudo-segments around each point annotation.
FSH-TAL architecture: SAM3 masks and point tracking, a multiple-instance few-shot classifier on trajectory-aligned tokens, and a temporal localizer with dilated convolutions, transformer blocks, and three decoder heads

Results

State of the art at every IoU threshold over varied supervision.

Average mAP, IoU 0.1–0.7

41.0%

+17.4 points over ActionFormer, fully supervised on the same pseudo-segments.

5-fold cross-validation over all 17 held-out recordings.

Per-Behavior Results

Well-defined behaviors localize best, even when they are rare.

Precisely defined behaviors beat loosely defined ones: Tilt and Peck are rare, yet among the best localized.

FSH-TAL beats every baseline on all six behaviors.

IoU 0.1–0.7, over the recordings that contain each behavior.

Cite

BibTeX

If you find this work useful, please cite it.

This citation is for the ECCV 2026 CV4Ecology workshop version. The WACV 2027 citation is coming soon.


@inproceedings{srinivasan2027automated,
  title = {Automated Temporal Localization of Behavior in Cichlid Fish with Few-Shot Trajectory Tokens},
  author = {Srinivasan, Bhargav and Lamousin, William and Liu, Jason and Phan, Charles and Osunbor, Ofure and Sharma, Shubh and McGeehan, Cade and Haroon, Tahir and Jayasuriya, Vijay and Parker, Coltan and Juntti, Scott and Kumar, Pulkit and Shrivastava, Abhinav},
  booktitle = {Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)},
  year = {2027},
  note = {To appear}
}