Franka teleop → training dataset

Franka FR3 51 episodes · 2.1 h RGB-D LeRobot format Safety stack

Driving the FR3 by hand

I teleoperate a Franka FR3 through a manipulation task — here, stacking rings on a peg — and record every run. Each run is one episode, saved with its camera frames and joint readings; the pile of episodes becomes a dataset a policy can learn from.

Nothing's scripted. The point is to capture how a person actually solves the task, mistakes and all.

Controller to arm, in RViz

Behind the demo: the controller's pose drives the arm through SE(3) transforms, mirrored live in RViz. Watching the model track the controller is how I check the mapping before anything moves the real FR3.

How it works

My commands don't reach the arm directly. A safety layer I wrote sits in between and clamps the workspace, end-effector speed, and command rate, so a bad input slows the robot instead of throwing it.
Every episode records synchronized RGB-D frames and joint states with timestamps, written in LeRobot's dataset format so it drops straight into standard imitation-learning training.
The dataset is the deliverable — 51 episodes, about 2.1 hours. Clean recording and honest labels matter more than a pretty trajectory, because a policy inherits whatever the data teaches it.

What broke, honestly

Reserved for the failure log: what broke while collecting this dataset and how it got fixed. Write-up in progress.

Links

LeRobot (dataset format)