Most existing activity datasets capture tasks from the outside: a fixed camera above a workbench, a third-person view of a robot arm. That perspective is useful for studying behavior from the outside. It is not useful for training the agent's own perception. A robot reaching for a cup needs to know what the cup looks like from its hand, not from a ceiling mount.
Ego-Centric Activity Data is a collection of first-person video and sensor streams of tasks performed in real environments. Every session is recorded from the performer's perspective, and every frame is annotated with the spatial and temporal context a model needs to close the perception-to-action loop: hand keypoints, object bounding boxes, contact events, task phase labels, and failure annotations where the demonstrator made a recoverable error.
What makes this different
Three things separate this dataset from the category:
- First-person perspective throughout: No overhead cameras, no external observers. Every recording is captured from the performer's point of view, giving the model the same visual input a deployed agent will have.
- Real environments, not lab setups: Sessions are collected in actual kitchens, laundry rooms, assembly areas, and outdoor settings - not purpose-built rigs. The lighting is inconsistent. The counters are cluttered. The objects are real objects with real variation in placement, condition, and occlusion.
- Per-frame annotation density: Raw footage is not a dataset. Every session is processed to produce frame-level annotation including hand keypoints, object labels and bounding boxes, contact events (when and where the hand makes contact with an object), task phase labels (reach, grasp, transport, place, release), and failure annotations for recoverable errors and recovery attempts.
Environments
Sessions are collected across three environment categories:
- Domestic: Kitchen tasks (food preparation, cooking, cleanup), laundry tasks (sorting, loading, folding), and general household tasks (organizing, cleaning, tool use around the home).
- Light industrial: Assembly tasks (joining components, fastening, fitting), packaging (box assembly, wrapping, sealing), and material handling (picking, placing, moving objects between stations).
- Outdoor unstructured: Navigation tasks in variable terrain, tool use in outdoor settings, and tasks that require maintaining spatial awareness across surfaces that are uneven, wet, or obstructed.
Task categories
Within each environment, sessions are organized into four task competencies:
- Manipulation: Grasping, placing, cutting, and assembling - tasks that require applying precise force to an object at a specific location.
- Articulation: Opening doors, drawers, and containers - tasks that require identifying a movable joint and applying directional force across its range of motion.
- Sorting and organizing: Folding, stacking, and arranging - tasks that require tracking multiple objects and placing them into a target configuration.
- Navigation: Moving between locations while maintaining awareness of obstacles, static structure, and dynamic elements in the environment.
Annotation schema
Each session produces a video file and a paired JSON annotation file. The annotation schema covers:
- Hand keypoints: Per-frame 2D skeletal annotations for both hands, with visibility confidence scores for occluded joints.
- Object bounding boxes: Per-frame bounding boxes for all task-relevant objects, labeled with category and instance ID.
- Contact events: Timestamped records of hand-object contact onset and offset, with contact type (grip, push, support) and estimated contact probability.
- Task phase labels: Per-frame phase label from the task phase taxonomy (reach, grasp, transport, place, release, idle) with transition timestamps.
- Failure annotations: Labeled segments where the demonstrator made a recoverable error, including error type, recovery action, and outcome. Failed attempts with successful recoveries are retained in the dataset - this is deliberate. A model trained only on clean demonstrations cannot learn to recover from errors.
Data specifications
- Modalities: RGB video (first-person), depth where available, IMU data
- Format: Video files with structured JSON annotation per session
- Annotation: Per-frame object labels, hand-object contact events, task phase labels, failure annotations
- First release: Domestic manipulation tasks - kitchen and laundry environments, early 2027
- Upcoming: Light industrial task collection begins 2027
categories
competencies
annotation types
throughout
Access
The first dataset release covers domestic manipulation tasks across kitchen and laundry environments and is expected in early 2027. Frontier labs and robotics teams interested in early access or co-collection partnerships can reach us at founders@generalmachines.ai.