Not every second of a video matters.
In a video of someone grilling, the flames and food carry the evidence for "barbecuing." The cat walking through the background probably does not.
That simple idea became the core of my senior thesis, which I defended this spring for High Honors in Computer Science at Dartmouth.
The question I started with was:
Can a vision-language model look at a video and identify which short moments actually carry evidence for the action? And if it can, do those labels make action recognition better?
To study this, I built Meaningful Moments: a dataset of about 4.58 million importance scores across roughly 500,000 videos and 622 action classes, spanning three standard video benchmarks: Something-Something v2, Kinetics-400, and Diving-48.
The answer was not simply yes or no. It depended on the dataset, and that was the interesting part.
- On Diving-48, where the action is concentrated in the dive itself, keeping the important moments preserved full-video accuracy. Keeping the "filler" segments collapsed performance by roughly 30 points.
- On Kinetics-400, aggressively keeping important segments outperformed uniform sampling.
- On Something-Something v2, the importance signal was real but did not recover full performance, because many classes depend on before-and-after state changes that the model did not always prioritize.
The broader takeaway
Temporal importance is useful, but only when the model's notion of "important" aligns with what the recognizer actually needs.
One engineering result I am especially proud of: scoring each segment directly matched a much slower iterative method while using roughly 25x fewer model calls. That efficiency is what made labeling half a million videos feasible.
None of this happened alone
My deepest thanks to my advisor, Professor SouYoung Jin, for her guidance and support throughout this thesis. Her courses and mentorship gave this work its foundation, and her feedback helped me work through the hardest parts.
Thank you as well to the SEE Lab for the weekly feedback that made the research sharper, and to my committee, Professor Alberto Quattrani Li and Professor Sami Saydjari, whose classes and comments shaped how I approach research and technical problems.
And to my family, especially my parents: thank you for the constant encouragement.
The dataset, code, and full thesis are public: dataset on Hugging Face, code on GitHub, and the full thesis (PDF).
If you work on video understanding or anything adjacent in AI, or just have questions about the research, I would genuinely like to hear what you think.
