Dual-arm VR teleoperation

2× Elite Robots CS66 125 Hz Cartesian servoing Quest 3, worn on the chest Real-time C++ Anchor/clutch control

Two arms, following my hands

Built at Nferent AI. Both arms follow my hands live from Meta Quest 3 controllers. Here they pass a block between grippers, with no trajectory planned in advance.

The Quest 3 hangs on my chest, cameras facing my hands. It still tracks the controllers, and I watch the real arms instead of a video feed.

It started with one arm

An Elite CS63 on one controller came first. Two arms need a process each, so a stall on one doesn't stutter the other, and their workspaces overlap right where the handoff happens.

How it works

A real-time C++ loop maps controller poses to end-effector targets through SE(3) transforms and servos in Cartesian space at 125 Hz. Hand pose arrives over UDP.
A clutch freezes the arms so I can re-grip mid-task, like lifting a mouse off the desk.
A safety layer clamps workspace, end-effector velocity and command rate.
When the headset loses a controller it reports a plausible wrong pose, so the loop drops frames it can't trust.

What made it hard

Near a wrist singularity the IK asks for huge joint speeds to make a small move, so the loop refuses instead of obeying.

My hand can go where the arm can't. The arm stops cleanly at its workspace edge instead of jerking.

Too little filtering and hand tremor becomes arm tremor. Too much and the arm moves like it's underwater.

Where it goes

The rig exists to collect demonstrations. The same loop drove the Franka FR3 for the ten-task dataset a π0.5 policy was fine-tuned on. How well I teleoperate caps what the policy can learn.