ROS 2 · SO-101
ROS 2 Command Reference
Full reference for the ROS 2 Jazzy pipeline on this Jetson (Ubuntu 24.04). Covers everything from first build to Human-in-the-Loop recording.
ROS 2 and the workspace auto-source via ~/.bashrc - no manual sourcing needed in new terminals.
Pre-flight checklist
Run the hardware sanity check every time before launching. Zero-responding servos means the arm's battery is off - not a USB or driver issue.
Build the workspace
Only needed after cloning fresh or wiping build/install. Regular YAML and launch file edits take effect immediately without a rebuild.
cd ~/ros2_ws colcon build --symlink-install --parallel-workers 4
If the build fails on a fresh clone (uv Python conflict), pin the system Python explicitly - CMake caches it after the first successful build so this flag is not needed again:
colcon build --symlink-install --parallel-workers 4 \ --cmake-args -DPython3_EXECUTABLE=/usr/bin/python3 -DPYTHON_EXECUTABLE=/usr/bin/python3
Hardware sanity check
Pings all 6 servos directly over the bus, bypassing ROS entirely. Run this before any launch when something seems off.
python3 -c "
import serial,time
for name,path in [('leader','/dev/so101_leader'),('follower','/dev/so101_follower')]:
s=serial.Serial(path,1000000,timeout=0.2)
ok=0
for i in range(1,7):
p=bytes([0xFF,0xFF,i,0x02,0x01]); p+=bytes([(~sum(p[2:]))&0xFF])
s.write(p); time.sleep(0.02)
if s.read(6): ok+=1
s.close()
print(name, f'{ok}/6 servos responding')
"All 6 responding = arm is powered and the bus is healthy. Zero responding = arm battery is off. A passing ping does not guarantee the arm is fully ready - a low battery can pass the ping but sag under the sustained current of initialization. If launch still fails right after a clean ping, charge before investigating further.
Recalibrate a single joint
Use this instead of a full recalibration when only one joint has drifted (e.g. the gripper). Leaves all other joints untouched.
pixi run -e lerobot python scripts/calibrate_single_joint.py # leader, gripper (default) pixi run -e lerobot python scripts/calibrate_single_joint.py --arm follower --motor wrist_roll
Requires an existing calibration file for that arm. Walks you through: move the joint to mid-range → Enter, sweep the full range → Enter.
Teleoperation
Standard teleoperation
ros2 launch so101_bringup teleop.launch.py
Common overrides
ros2 launch so101_bringup teleop.launch.py use_cameras:=false # skip cameras - faster startup ros2 launch so101_bringup teleop.launch.py use_teleop_rviz:=false # skip RViz ros2 launch so101_bringup teleop.launch.py arm_controller:=trajectory_controller # smoother motion via trajectory interpolation ros2 launch so101_bringup teleop.launch.py use_rerun:=true use_teleop_rviz:=false # Rerun visualizer instead of RViz
Parameter reference
| Parameter | What to change it to |
|---|---|
| use_cameras:=false | Skip camera nodes - useful for fast hardware checks or when cameras aren't needed |
| use_teleop_rviz:=false | Skip RViz - saves resources, use alongside use_rerun:=true if you want visualization |
| arm_controller:=trajectory_controller | Smoother motion via trajectory interpolation instead of raw position setpoints - used by MoveIt |
| use_rerun:=true | Use Rerun visualizer instead of RViz - pair with use_teleop_rviz:=false |
Recording episodes
Recording requires two terminals running simultaneously.
Terminal 1 - Start the recording session
ros2 launch so101_bringup recording_session.launch.py \ experiment_name:=pick_and_place \ task:="Pick up the cube and place it in the container." \ use_rerun:=true
Parameter reference
| Parameter | What to change it to |
|---|---|
| experiment_name | Folder name for this session - episodes land in ~/.ros/so101_episodes/<experiment_name>/ |
| task | Plain English description of the task - stored in the episode metadata |
| use_rerun:=true | Open Rerun visualizer to monitor cameras and joint state live during recording |
Terminal 2 - Keyboard episode controls
ros2 run episode_recorder teleop_episode_keyboard
r - start recording
s - save and stop episode
d / Backspace - discard episode
q - quit
h - help
Episodes are saved as MCAP rosbags at ~/.ros/so101_episodes/<experiment_name>/episode_NNNNNN/
Review recorded episodes in the browser
pixi run python scripts/so101_episode_viewer_mcap.py --episodes_root ~/.ros/so101_episodes/pick_and_place
Vary the object's starting position slightly between every episode. Dataset diversity is the single biggest factor in preventing out-of-distribution freezing during inference.
Convert & upload to Hugging Face
One-time authentication
pixi run -e lerobot -- hf auth login pixi run -e lerobot -- hf auth whoami
Convert rosbags and push to the Hub
pixi run -e lerobot convert -- \ --input-dir ~/.ros/so101_episodes/pick_and_place \ --config ~/ros2_ws/src/so101-ros-physical-ai/rosbag_to_lerobot/config/so101.yaml \ --repo-id <hf-username>/so101-pick-and-place \ --push-hub
Parameter reference
| Parameter | What to change it to |
|---|---|
| --input-dir | Path to your recorded experiment folder under ~/.ros/so101_episodes/ |
| --config | so101.yaml for joint-space (default), so101_cartesian.yaml for Cartesian pose - see section 05 |
| --repo-id | Your HuggingFace username/dataset-name - the dataset is private by default |
| --push-hub | Omit this flag to convert locally only without uploading |
Local-only conversion (no upload)
pixi run -e lerobot convert -- \ --input-dir ~/.ros/so101_episodes/pick_and_place \ --config ~/ros2_ws/src/so101-ros-physical-ai/rosbag_to_lerobot/config/so101.yaml \ --repo-id local/so101_test
Visualize a converted dataset
pixi shell -e lerobot lerobot-dataset-viz --repo-id local/so101_test --episode-index 0
Conversion runs at ~4x faster than the old path thanks to streaming video encoding - no intermediate PNG files on disk. Use --no-streaming-encode to fall back to the old path if something goes wrong.
Training
pixi shell -e lerobot lerobot-train \ --dataset.repo_id=<hf-username>/so101-pick-and-place \ --policy.type=act \ --output_dir=outputs/train/act_so101_pick_place \ --job_name=act_so101_pick_place \ --policy.device=cuda
Parameter reference
| Parameter | What to change it to |
|---|---|
| --dataset.repo_id | The HuggingFace dataset you converted and uploaded in section 03 |
| --policy.type | act for ACT, smolvla for SmolVLA - smolvla is significantly slower to train |
| --output_dir | Local folder where checkpoints are saved during training |
| --policy.device | cuda to use the Jetson GPU - always use this, CPU training is impractically slow |
Inference
Inference always requires two terminals.
Terminal 1 - Bring up the follower and cameras
ros2 launch so101_bringup inference.launch.py
Terminal 2 - Run the policy
cd ~/ros2_ws/src/so101-ros-physical-ai # ACT policy (joint-space, default) pixi run -e lerobot infer -- --ros-args -p repo_id:="<hf-username>/<act-policy>" # SmolVLA policy (joint-space) pixi run -e lerobot infer -- --ros-args \ -p repo_id:="<hf-username>/<smolvla-policy>" -p policy_type:=smolvla \ -p camera_top_name:=camera1 -p camera_wrist_name:=camera2 # ACT policy trained on Cartesian pose data (so101_cartesian.yaml dataset) pixi run -e lerobot infer -- --ros-args \ -p repo_id:="<hf-username>/<cartesian-act-policy>" -p use_cartesian_action:=true # Specific checkpoint branch instead of main pixi run -e lerobot infer -- --ros-args \ -p repo_id:="<hf-username>/<policy>" -p revision:=step_020000 -p use_cartesian_action:=true # Async inference - policy runs on a remote GPU, arm runs locally pixi run -e lerobot async_infer -- --ros-args \ -p repo_id:="<hf-username>/<policy>" -p policy_type:=smolvla \ -p server_address:=<gpu-host>:8090 -p fps:=50.0 \ -p camera_top_name:=camera1 -p camera_wrist_name:=camera2
Parameter reference
| Parameter | What to change it to |
|---|---|
| repo_id | HuggingFace model repo (username/model-name) |
| policy_type | act (default) or smolvla - must match the architecture the model was trained with |
| use_cartesian_action | Only set to true for a policy trained with so101_cartesian.yaml. Never set this for a joint-space policy |
| revision | Checkpoint branch to load (e.g. step_020000). Omit to use main. Check the model's HF page - main is sometimes empty |
| camera_top_name / camera_wrist_name | Remap camera keys if the policy was trained with different camera names. Check the model's config.json input_features |
| server_address | Async only - host:port of the remote GPU policy server |
Killing the ROS inference node does NOT release servo torque. The STS3215 servos hold their last commanded position independently. If the arm is straining after a kill, cut power to the arm directly - do not relaunch first.
Image resolution mismatches are handled automatically - frames are resized to match whatever resolution the policy was trained on. Loading a policy successfully is not proof it is safe to run. Watch any unfamiliar policy closely on the first run with a hand near the power switch.
Cartesian pose pipeline
An alternative to raw joint angles where the model sees and predicts end-effector position + rotation vector + gripper. The conversion from joint angles to Cartesian pose happens at dataset conversion time - just swap the config flag during convert:
pixi run -e lerobot convert -- \ --input-dir ~/.ros/so101_episodes/pick_and_place \ --config ~/ros2_ws/src/so101-ros-physical-ai/rosbag_to_lerobot/config/so101_cartesian.yaml \ --repo-id local/so101_cartesian_test
Recording is unchanged - you use the exact same recording workflow. The Cartesian transform runs on already-saved rosbags. Train normally, then add use_cartesian_action:=true at inference time.
Human-in-the-Loop (HIL) recording
A policy runs autonomously on the follower while you can grab the leader at any moment to take over, correct, or complete the task. The full trajectory - autonomous and human sections blended - records as one normal episode.
Terminal 1 - Start the HIL session
ros2 launch so101_bringup hil_recording_session.launch.py \ experiment_name:=hil_pick_and_place \ task:="Pick up the red block and place it in the box." \ repo_id:=<hf-username>/<policy> \ revision:=step_020000 \ use_cartesian_action:=true
Parameter reference
| Parameter | What to change it to |
|---|---|
| experiment_name | Folder name for HIL episodes - same structure as regular recording |
| task | Plain English task description stored in episode metadata |
| repo_id | The policy to run autonomously - same format as inference |
| revision | Checkpoint branch - omit to use main |
| use_cartesian_action | Only for a policy trained with so101_cartesian.yaml - omit for joint-space policies |
Terminal 2 - Episode controls + handoff key
ros2 run episode_recorder teleop_episode_keyboard
r - start recording
s - save and stop episode
d / Backspace - discard episode
q - quit
y - toggle control between policy and human
When pressing y to hand control to the human, the leader arm moves under its own power for ~2 seconds to sync its pose to match the follower. Stand clear of the leader during this window - it is genuinely torqued and moving.
Press y again to hand control back to the policy. The policy resumes cleanly from wherever the follower is - no hand-off logic needed since inference is already closed-loop over live observations.
Troubleshooting
Kill all SO-101 ROS processes
Use when a launch is stuck or half-broken. robot_state_publisher is easy to forget - leftover instances silently serve a stale robot_description to the next launch.
pkill -9 -f ros2_control_node; pkill -9 -f gscam_node; pkill -9 -f rviz2; pkill -9 -f robot_state_publisher
Check what is running
pgrep -af "gscam_node|rviz2|ros2_control_node|robot_state_publisher|static_transform_publisher"
Check topic rates
ros2 topic hz /follower/image_raw ros2 topic hz /static_camera/image_raw ros2 topic list
Camera and USB device management
v4l2-ctl --list-devices # list connected cameras and their /dev/videoN nodes ls -l /dev/so101_leader /dev/so101_follower /dev/cam_wrist /dev/cam_overhead # Grab a still frame to visually identify a camera (no ROS needed) ffmpeg -y -f v4l2 -input_format mjpeg -video_size 1280x720 -i /dev/videoN -frames:v 1 -update 1 out.jpg # Reload udev rules after editing /etc/udev/rules.d/99-so101.rules sudo udevadm control --reload-rules && sudo udevadm trigger # Kernel-level USB errors (camera dropouts, bandwidth issues) sudo dmesg | tail -60
Cameras are identified by physical USB port (ID_PATH), not by serial number - all three cameras are the same model. Keep each camera in its assigned port or update the udev rule to match the new port.
Known gotchas on this machine
USB bandwidth
All 5 devices (2 arms + 3 cameras) share one 480 Mbps USB 2.0 bus. Three simultaneous camera streams exceeds the link - 2 cameras run reliably. A 3rd camera should go on the unused USB 3 controller (Bus 002), with a self-powered hub.
CUDA on Jetson
pip's default torch for aarch64 ships without CUDA. The fix is pinned in pixi.toml (cu129 build). cu130 passes torch.cuda.is_available() but crashes on the first real transformer forward pass. Always verify with an actual model forward pass after any torch version change.
Dead servo bus
SerialPort read timeout on all servos = arm battery is off, not a cable issue. The USB-serial adapter enumerates fine on USB logic power regardless of whether the arm is powered.
Servo torque after kill
STS3215 servos hold their last commanded goal position even after the host process dies. kill -9 cannot run a graceful torque-disable. If the arm is straining, cut power to it directly.
Using the original LeRobot pipeline instead?
See the bare LeRobot command reference for the pre-ROS 2 workflow.