Experimenting with egocentric video
Hey guys,
With robotics growing so fast, first-person (egocentric) vision is becoming a massive domain in CV on its own. If robots are ever going to help us in the real world, they need to understand how humans handle objects from our own perspective.
I've been deep in experimentation mode and performing some test with CV model on egocentric video from scratch on everyday simple tasks (annotation -> model training -> implementation)!
For this project, I focused on a simple, everyday task: opening and closing a bottle cap. Here is a quick look at the video showing the real-time tracking and state changes in action:
- Data Annotation: I started by capturing raw egocentric footage. To get clean bounding boxes for the bottle and cap across the sequence, I used Labellerr. It made handling the frame-by-frame labeling smooth and kept the dataset precise.
- Model Training & Tracking: I paired object detection for the assets (bottle and cap) with hand skeleton tracking to map exactly how the fingers grasp and interact with the objects.
- State Logic Building: Once the spatial coordinates were tracking properly, I built a custom state machine logic on top of it. The system actively differentiates between IDLE, OPENING THE BOTTLE, and CLOSING THE BOTTLE based on hand-to-object intersections and hand velocity.
This is one of many examples i am experimenting with egocentric video (feel free to suggest some ideas regarding it)
Would love to hear your thoughts! Are any of you working on egocentric datasets or robotics perception pipelines right now? What are the biggest bottlenecks you’re running into with first-person data?