



What the robot sees: frames from the training recordings, overhead and wrist cameras.
The challenge
The Robotekin LeRobot Hackathon 2026 gave teams a weekend to build a working robotics prototype with Hugging Face LeRobot on SO-101 arms, around object manipulation, robot vision, imitation learning and social applications.
We took a task from industrial inspection and combined the first three: pick a part, check it, put it in the right place. The part is a pressed metal bracket; some carry a pin and some do not. Its shape means the right grasp depends on how it lies in the tray, so picking is learned as its own policy, separate from the sorting motions.
How it works
One application runs the whole loop on the real arm. Three ACT policies move the robot, one classifier makes the decision, and a spoken order sets the rule.
Order
Speech in Spanish, English or Basque becomes a sorting rule.
bin-intent-gliner2
Pick
Once a part lies still in the tray, the arm grasps it and lifts it to the camera.
act-pick
Inspect
1.5 s of overhead and wrist frames are averaged: pin or no pin.
pin-classifier
Decide
The order maps the verdict to a box. A part with no matching rule is held until one arrives.

Green box
Parts the order sends here (usually: with pin). Carry, release, back to rest.
act-up-down-good
White basket
Parts the order sends here (usually: no pin). Carry, release, back to rest.
act-up-down-badThe loop then waits for the next part. A pick that fails is retried up to three times. The
sorting policies only know where to carry a part, so a new rule needs no retraining. Everything runs in
30_full_demo_app.py.
The cell
| Component | Description |
|---|---|
| Arm | SO-101 follower, six joints: shoulder pan, shoulder lift, elbow flex, wrist flex, wrist roll and gripper. Arm joints in degrees, gripper 0–100. |
| Teleoperation | SO-101 leader arm. During recording it drives the follower; its position is the action the policies learn to predict. |
| Cameras | Two USB RGB cameras at 640×480 and 30 fps: front, on a tripod above the tray, and wrist, on the gripper. |
| Work area | Orange tray with the parts, inspection position under the overhead camera, green box and white basket. |
| Software | LeRobot (dataset format v3.0), PyTorch, faster-whisper, GLiNER2. |
Teaching the robot
Demonstrations. An operator guided the leader arm while LeRobot recorded both cameras, the follower's joints and the leader's commands at 30 fps. One recording tool covered the three motions, one key each: pick, carry to the green box, carry to the white basket. Between a pick and a carry the operator looked at the part and chose the box. Only accepted takes were kept.
Policies. The three motion policies use ACT (Action Chunking with Transformers): a ResNet18 image encoder and a transformer that predicts a chunk of future joint targets from the images and the joint state; the robot executes part of the chunk and replans. Each was trained over several rounds that changed one thing at a time (data filters, chunk length, an extra velocity input) and scored on held-out episodes against "hold still" and "repeat the state".
Classifier. An ImageNet-pretrained EfficientNet-B0, shared by both cameras, looks at the held part and answers pin or no pin. About 85 % accurate on recording sessions it never saw.
Models
| Stage | Model | Method | Offline result |
|---|---|---|---|
| Pick | act-pick | ACT, 9 training rounds | Action error 27 % below holding still |
| Inspect | pin-classifier | EfficientNet-B0, two views | ≈ 85 % accuracy on unseen sessions |
| Green box | act-up-down-good | ACT, 6 versions | 16 % better than repeating the state |
| White basket | act-up-down-bad | ACT with joint-velocity input | 14 of 14 releases correct on held-out episodes |
Datasets
| Stage | Dataset | Content | Size |
|---|---|---|---|
| Pick | robotekin-so101-pick | Pick the part and hold it up to the camera | 350 episodes |
| Inspect | robotekin-so101-pin-images | Overhead and wrist image pairs, pin or no pin | 1,547 pairs |
| Green box | robotekin-so101-green-box | Carry the part to the green box and release it | 189 episodes |
| White basket | robotekin-so101-white-box | Carry the part to the white basket and release it | 153 episodes |
The motion datasets are LeRobot v3.0 and join the per-session recordings; each lists its sources in meta/sources.json.
Limitations
- The model metrics are offline comparisons with the demonstrations; they do not count how many parts reach the right box.
- All data come from one room, one lighting setup and two days of recording. The classifier has not been tested under other conditions or on another batch of parts.
- The sorting policies start from the pose in which the pick policy leaves the part; poses far from the demonstrations are outside what they learned.
Made by
Iker Azurmendi · Mikel Zulueta · Ander Autillo · Diego Suarez
Built during the Robotekin LeRobot Hackathon 2026 in Vitoria-Gasteiz. Thanks to Bravo Robots and Robotekin for the hackathon, to Hugging Face for LeRobot, and to everyone who took part in the event.