Cygnus Robotics

Your labels say success. The footage says otherwise.

We reconstruct attempts, recoveries, and idle tails across robotics and AV datasets in multiple formats, then cut training sets to what is actually worth learning from.

Run on your clipMeasured results

ep0 annotated · green = kept · red = dropped · 21.5% removed

3/3
Episodes with hidden failure
26.4%
Frames cut (full task arc)
88%
Cut if success-only
141
fps end-to-end (laptop CPU)

01 · The problem

Every frame labelled “Grab the Lego and store it in red box”

A single task string is stamped across the whole episode. Failures, re-grasps, and post-success idle look like good demonstrations to any model trained on the raw labels.

Stacked bars showing each episode broken into attempts, recoveries, success, and idle

Label on disk

“Grab the Lego and store it in red box”

Applied to all 596 frames of episode 0.

Structure recovered

  • Failed attempt → recovery → failed attempt → recovery → success.
  • 161 idle-tail frames across 3 episodes after the task was done.
  • 37 MB source → 3.6 MB curated under “full task arc”.

02 · Measured results

1,619 frames · 3 episodes · every cut auditable

Annotated videos show keep/drop per frame. Phase rows below are the same numbers the pipeline writes to sidecars.

EpisodeFramesKeptDroppedReducedAttemptsRecoveriesIdle tailSource → curated
ep059646812821.5%325412.6 → 1.4 MB
ep145831314531.7%326610.8 → 1.1 MB
ep256541115427.3%324114.0 → 1.1 MB

Episode 0

596f to 468f · 21.5% removed · 19.9s to 15.6s

Phase timeline

  • Attempt 1 · 0.00 to 3.37s · 101f · kept 45
  • Recovery after a drop or miss · 3.37 to 14.00s · 319f · kept 304
  • Attempt 2 · 14.00 to 14.43s · 13f · kept 13
  • Recovery after a drop or miss · 14.43 to 15.30s · 26f · kept 24
  • Attempt 3: success · 15.30 to 18.07s · 83f · kept 82
  • Idle tail after the task was done · 18.07 to 19.87s · 54f · kept 0

Att

3

Rec

2

Idle

54

Storage

12.6 MB to 1.4 MB

  • The episode label describes a clean run, but the footage contains failure and recovery.
  • The labelled span runs past the moment the task was finished.

Episode 1

458f to 313f · 31.7% removed · 15.3s to 10.4s

Phase timeline

  • Attempt 1 · 0.00 to 4.40s · 132f · kept 58
  • Recovery after a drop or miss · 4.40 to 9.73s · 160f · kept 155
  • Attempt 2 · 9.73 to 10.60s · 26f · kept 26
  • Recovery after a drop or miss · 10.60 to 11.67s · 32f · kept 32
  • Attempt 3: success · 11.67 to 13.07s · 42f · kept 42
  • Idle tail after the task was done · 13.07 to 15.27s · 66f · kept 0

Att

3

Rec

2

Idle

66

Storage

10.8 MB to 1.1 MB

  • The episode label describes a clean run, but the footage contains failure and recovery.
  • The labelled span runs past the moment the task was finished.

Episode 2

565f to 411f · 27.3% removed · 18.8s to 13.7s

Phase timeline

  • Attempt 1 · 0.00 to 3.37s · 101f · kept 42
  • Recovery after a drop or miss · 3.37 to 12.13s · 263f · kept 235
  • Attempt 2 · 12.13 to 13.53s · 42f · kept 42
  • Recovery after a drop or miss · 13.53 to 14.27s · 22f · kept 21
  • Attempt 3: success · 14.27 to 17.47s · 96f · kept 71
  • Idle tail after the task was done · 17.47 to 18.83s · 41f · kept 0

Att

3

Rec

2

Idle

41

Storage

14.0 MB to 1.1 MB

  • The episode label describes a clean run, but the footage contains failure and recovery.
  • The labelled span runs past the moment the task was finished.
Kept versus dropped frames per episode
kept_vs_dropped.png · include_recovery policy
Pipeline throughput by stage
throughput.png · Apple M1 laptop CPU, no GPU
StageFPSEmbedder
Decode video211local
Embed (fast visual)458local
End to end (fast visual)141local
Embed (foundation model)16.6clip
End to end (foundation model)15.4clip

Full 23,041-frame dataset ≈ 3 min single-threaded on Apple M1 laptop CPU, no GPU. Source: LeRobot worldwide hackathon: Quarter Brain lego picking· camera side · 30 fps · 46 episodes.

03 · Policies

Same footage, three training goals

Curation is an input. What counts as redundant depends on whether you need recoveries for robustness or only clean successes for imitation.

Frames kept under three curation policies

Deduplicate only

1236f · −23.7%

Remove visual redundancy only. Task structure stays unchanged, making this a safe default when the training goal is still unclear.

Keep the full task arc

1192f · −26.4%

Keep failures and recoveries; drop post-success idle. Policies that never see a drop cannot learn to recover.

Successful demonstration only

195f · −88%

Keep only the attempt that worked. This aggressive cut is designed for clean imitation and delivers the largest storage savings.

04 · AV smoke · L2D R0

Same pipeline on open driving data

yaak-ai/L2D @ R0, episode 0, observation.images.front_left. Task: “Drive straight up to the yield sign and take first exit from the roundabout”. Event detection labelled the run recovery; success_only curation kept 33/150 frames (−78%).

6
Progress backtracks
7
Estimated attempts
78%
Frames removed (success_only)
33/150
Frames kept
annotated · keep/drop overlay · 78% removed
curated · survivors only · 33 frames @ 10 fps

Reproduce locally with python scripts/smoke_l2d.py in yotta-core (selective R0 download). Full archive: yaak-ai/L2D.

05 · Open datasets

Where to point the pipeline next

The manipulation demo is shipped. For autonomous driving, start with LeRobot-native L2D, then convert comma2k19 or nuScenes if you need more diversity.

FitDatasetFormatDomainScaleWhy
Best fitL2D: Learning to Drive (Yaak × Hugging Face)

HF blog

LeRobotDataset (v2.1 / v3 streaming)Autonomous / student and expert driving~5,000 hours · R2+ community releases · full archive >1 PBNative LeRobot format with cameras + vehicle state/actions. It is the best first AV target for the Cygnus ingest path, with no format converter required.
Needs convertcomma2k19Custom (needs conversion)Highway / openpilot-style driving~33 hours · ~100 GB classSmaller, well-documented open driving logs. Convert camera + CAN into LeRobot episodes to stress quality and motion↔video lag gates.
Needs convertnuScenesCustom (needs conversion)Urban AV perception1,000 scenes · ~5.5 hours labeledStandard multi-camera urban scenes. Strong for occlusion and lighting QA once wrapped as time-aligned episodes.
On this siteQuarter Brain lego picking (shipped on this site)LeRobotManipulation teleop23,041 frames · 46 episodesMeasured demo below: 46 episodes, with every sampled run hiding failure and recovery under a clean task label.

Recommended first AV pull: stream a small L2D revision with StreamingLeRobotDataset("yaak-ai/L2D"), run quality + event detection, then scale. See LeRobot goes to driving school.

06 · Live demo

Run the pipeline on your own clip

Upload a short robot episode. You get failure/recovery detection, proposed phase labels, lighting/freeze QA, and a curated cut. Uploads and results are deleted about an hour after processing.

07 · Pipeline

Nine stages, adoptable one at a time

Each stage writes a sidecar next to the untouched recording, so you can adopt one gate without buying the whole stack.

  1. 01Ingest and normalizeHF datasets → reviewable timeline of trajectories + camera.
  2. 02Label scaffoldingTime-ranged annotation templates beside immutable recordings.
  3. 03Action-state inferenceOpen-vocabulary metadata over frame windows.
  4. 04Geometry enrichmentDepth / normals / segmentation sidecars.
  5. 05Quality assessmentExemplar retrieval, defect gates, trajectory matching.
  6. 06Frame curationVisual drift + kinematics so real motion is not treated as redundancy.
  7. 07Failure detectionDrops, re-grasps, backtracks, and post-success idle, with no per-frame labels required.
  8. 08Annotation QAFlag lying labels; propose corrected phase spans.
  9. 09Training handoffPruned datasets, curated video, lossless state/action round-trip.