SO-101 Pin Sorting
Robotekin LeRobot Hackathon · Vitoria-Gasteiz · Oct 2026

Pick it. Check it. Sort it.

An SO-101 robot arm picks a metal bracket from a tray, a vision model checks whether it carries a pin, and a learned policy drops it in the green box or the white basket. Every motion was taught by demonstration during one hackathon weekend.

Iker Azurmendi · Mikel Zulueta · Ander Autillo · Diego Suarez

Robotekin LeRobot Hackathon 2026, Robotory, Vitoria-Gasteiz

Live demo, 25 October 2026: parts picked from the orange tray and sorted into the green box and the white basket.
Overhead view: the arm grasping a bracket in the orange tray
overhead · pick
Overhead view: the bracket held up above the tray for inspection
overhead · inspect
Wrist view: the bracket carried over the green box
wrist · green box
Wrist view: the bracket carried towards the white basket
wrist · white basket

What the robot sees: frames from the training recordings, overhead and wrist cameras.

01

The challenge

The Robotekin LeRobot Hackathon 2026 gave teams a weekend to build a working robotics prototype with Hugging Face LeRobot on SO-101 arms, around object manipulation, robot vision, imitation learning and social applications.

We took a task from industrial inspection and combined the first three: pick a part, check it, put it in the right place. The part is a pressed metal bracket; some carry a pin and some do not. Its shape means the right grasp depends on how it lies in the tray, so picking is learned as its own policy, separate from the sorting motions.

What "good" and "bad" mean here. The words name the part, never the quality of a demonstration: every training episode is a successful one. The usual rule:
good = with pin → green boxbad = no pin → white basket
Wrist camera close-up of a bracket with its threaded pin
A bracket with its pin, seen by the wrist camera as the gripper approaches.
02

How it works

One application runs the whole loop on the real arm. Three ACT policies move the robot, one classifier makes the decision, and a spoken order sets the rule.

“Parts with a pin go to the green box”voice · text · Telegram
01

Order

Speech in Spanish, English or Basque becomes a sorting rule.

bin-intent-gliner2
02

Pick

Once a part lies still in the tray, the arm grasps it and lifts it to the camera.

act-pick
03

Inspect

1.5 s of overhead and wrist frames are averaged: pin or no pin.

pin-classifier
04

Decide

The order maps the verdict to a box. A part with no matching rule is held until one arrives.

05a

Green box

Parts the order sends here (usually: with pin). Carry, release, back to rest.

act-up-down-good
05b

White basket

Parts the order sends here (usually: no pin). Carry, release, back to rest.

act-up-down-bad

The loop then waits for the next part. A pick that fails is retried up to three times. The sorting policies only know where to carry a part, so a new rule needs no retraining. Everything runs in 30_full_demo_app.py.

03

The cell

ComponentDescription
ArmSO-101 follower, six joints: shoulder pan, shoulder lift, elbow flex, wrist flex, wrist roll and gripper. Arm joints in degrees, gripper 0–100.
TeleoperationSO-101 leader arm. During recording it drives the follower; its position is the action the policies learn to predict.
CamerasTwo USB RGB cameras at 640×480 and 30 fps: front, on a tripod above the tray, and wrist, on the gripper.
Work areaOrange tray with the parts, inspection position under the overhead camera, green box and white basket.
SoftwareLeRobot (dataset format v3.0), PyTorch, faster-whisper, GLiNER2.
04

Teaching the robot

692teleoperated episodes
1,547labelled image pairs
3ACT policies
2days of recording

Demonstrations. An operator guided the leader arm while LeRobot recorded both cameras, the follower's joints and the leader's commands at 30 fps. One recording tool covered the three motions, one key each: pick, carry to the green box, carry to the white basket. Between a pick and a carry the operator looked at the part and chose the box. Only accepted takes were kept.

Policies. The three motion policies use ACT (Action Chunking with Transformers): a ResNet18 image encoder and a transformer that predicts a chunk of future joint targets from the images and the joint state; the robot executes part of the chunk and replans. Each was trained over several rounds that changed one thing at a time (data filters, chunk length, an extra velocity input) and scored on held-out episodes against "hold still" and "repeat the state".

Classifier. An ImageNet-pretrained EfficientNet-B0, shared by both cameras, looks at the held part and answers pin or no pin. About 85 % accurate on recording sessions it never saw.

Overhead view of a held bracket Wrist view of the same bracket
PINoverhead + wrist
05

Models

StageModelMethodOffline result
Pickact-pickACT, 9 training roundsAction error 27 % below holding still
Inspectpin-classifierEfficientNet-B0, two views≈ 85 % accuracy on unseen sessions
Green boxact-up-down-goodACT, 6 versions16 % better than repeating the state
White basketact-up-down-badACT with joint-velocity input14 of 14 releases correct on held-out episodes
06

Datasets

StageDatasetContentSize
Pickrobotekin-so101-pickPick the part and hold it up to the camera350 episodes
Inspectrobotekin-so101-pin-imagesOverhead and wrist image pairs, pin or no pin1,547 pairs
Green boxrobotekin-so101-green-boxCarry the part to the green box and release it189 episodes
White basketrobotekin-so101-white-boxCarry the part to the white basket and release it153 episodes

The motion datasets are LeRobot v3.0 and join the per-session recordings; each lists its sources in meta/sources.json.

07

Limitations

Made by

Iker Azurmendi · Mikel Zulueta · Ander Autillo · Diego Suarez

Built during the Robotekin LeRobot Hackathon 2026 in Vitoria-Gasteiz. Thanks to Bravo Robots and Robotekin for the hackathon, to Hugging Face for LeRobot, and to everyone who took part in the event.