Post
363
**Training ACT on SO-101: From Woodpecker to 90% Success (All the Mistakes Included)**
I spent 3 weeks training Action Chunking Transformer on SO-101 for pick-and-place. Spoiler: the first attempt trained a woodpecker that just pecked the table. 🐦
**What's Different About This Post:**
Most ACT tutorials show the success. I documented every failure, hardware issue, and debugging step. If you're new to SO-101/LeRobot/ACT, hopefully my mistakes save you time.
Try 1: The Woodpecker
- Followed the LeRobot tutorial, collected 50 episodes
- Beautiful loss curves ✅
- Robot learned to peck at table ❌
- Rookie mistakes: moving cameras, arm calibration mismatch, limited data diversity, looking at follower arm during teleop (it's cheating!)
Try 2: Engineering Upgrades
- Fixed hardware setup (tape + markers everywhere)
- USB udev rules for camera stability
- Formal task definition with stratified sampling
- Built proper eval pipeline with progress scoring
- Motor breakdown mid-collection (broke the gripper with excessive force 💀)
- Results: 60% in-distribution success, 10% OOD (better, but not great)
Try 3: More & Better Data
sherryxychen/2025-09-07_act
- 125 episodes with rotation variations ( sherryxychen/2025-09-01_pick-and-place-block)
- Better workspace coverage
- Improved grasping technique
- Results: 90% in-distribution success, 75% OOD! 🎉
Key Learnings:
- Consistent hardware setup is everything
- Don't look at the follower arm during teleop
- Data diversity is key for generalization
- Debug infrastructure matters
- Real robots break in mysterious ways (buy spare motors!)
Full write-up (with more videos!): https://huggingface.co/blog/sherryxychen/train-act-on-so-101
Code: https://github.com/sherrychen1120/so101_bench
Happy to answer any questions! 🤗
#imitation-learning #lerobot #act #so-101
I spent 3 weeks training Action Chunking Transformer on SO-101 for pick-and-place. Spoiler: the first attempt trained a woodpecker that just pecked the table. 🐦
**What's Different About This Post:**
Most ACT tutorials show the success. I documented every failure, hardware issue, and debugging step. If you're new to SO-101/LeRobot/ACT, hopefully my mistakes save you time.
Try 1: The Woodpecker
- Followed the LeRobot tutorial, collected 50 episodes
- Beautiful loss curves ✅
- Robot learned to peck at table ❌
- Rookie mistakes: moving cameras, arm calibration mismatch, limited data diversity, looking at follower arm during teleop (it's cheating!)
Try 2: Engineering Upgrades
- Fixed hardware setup (tape + markers everywhere)
- USB udev rules for camera stability
- Formal task definition with stratified sampling
- Built proper eval pipeline with progress scoring
- Motor breakdown mid-collection (broke the gripper with excessive force 💀)
- Results: 60% in-distribution success, 10% OOD (better, but not great)
Try 3: More & Better Data
sherryxychen/2025-09-07_act
- 125 episodes with rotation variations ( sherryxychen/2025-09-01_pick-and-place-block)
- Better workspace coverage
- Improved grasping technique
- Results: 90% in-distribution success, 75% OOD! 🎉
Key Learnings:
- Consistent hardware setup is everything
- Don't look at the follower arm during teleop
- Data diversity is key for generalization
- Debug infrastructure matters
- Real robots break in mysterious ways (buy spare motors!)
Full write-up (with more videos!): https://huggingface.co/blog/sherryxychen/train-act-on-so-101
Code: https://github.com/sherrychen1120/so101_bench
Happy to answer any questions! 🤗
#imitation-learning #lerobot #act #so-101