Teleop data → a π0.5 policy

π0.5 VLA 10 tasks CS66 + FR3 125 Hz teleop LeRobot format

Driving the FR3 by hand

Teleoperating a Franka FR3 to stack rings on a peg. Each run is recorded as an episode, with camera frames and joint readings, and the episodes become the training set.

Mistakes stay in. A policy trained only on clean demonstrations has never seen a recovery.

Controller to arm, in RViz

The controller's pose drives the arm through SE(3) transforms, mirrored live in RViz. A flipped axis is obvious there and expensive on the real FR3, so I check the mapping before it moves.

The loop that collects it

The same real-time C++ loop as the CS66 pair: Cartesian servoing at 125 Hz, Quest 3 hand pose over UDP.
Poses the headset can't vouch for are dropped.
The same safety clamps apply. A bad run here also ends up in the training set.
Ten tasks across both arms, so the policy doesn't fit one arm's quirks.

Training the policy

I fine-tuned a π0.5 vision-language-action policy on the 10-task set and worked on its inference path.

The data is in LeRobot's format, so it trains without a custom conversion step.

What I would fix

Ten tasks is enough to fine-tune on, not enough to claim generalisation.
I didn't log episode counts per task. Next time I'd fix what the collection script writes before collecting.