Week of August 11, 2026
HIL recording working; don't run unverified policies unattended
HIL recording is working. A policy runs the follower on its own and you can grab the leader at any point to take over - everything records as one continuous episode. When you press the toggle, the follower freezes, the leader moves under its own power to match the follower's position (about 2 seconds, keep your hands clear), then goes limp so you can drive it. Press again and the policy takes back over from wherever the arm is. Tested with ThavT/SO101PickPlaceCartesian (step_020000) and the whole cycle worked. Also tried running a policy Tony trained on a different task without ROS - it loaded fine but immediately drove the arm into itself. A policy loading without errors does not mean it's safe. If the arm is straining after you kill the node, cut the power directly. Killing the process does not release servo torque.
Week of August 4, 2026
Cartesian pose pipeline working after IK fix
Tried training with Cartesian pose instead of raw joint angles. The idea is that giving the model end-effector position rather than joint values might help it generalize better. The conversion from joints to Cartesian pose happens at dataset conversion time using forward kinematics on already-recorded rosbags, so the recording workflow is unchanged. First inference run kept throwing IK errors on almost every tick. The solver was rejecting targets because the implied velocity per tick was over the joint limits. Disabled velocity limits in the IK config since the servos control physical speed anyway - the solver check was doing nothing useful. ThavT/SO101PickPlaceCartesian (step_020000 and step_014000) both ran cleanly after that. Continued recording SmolVLA datasets alongside.
Week of July 28, 2026
SmolVLA goes from 70s to basically instant after CUDA fix
SmolVLA was taking 70 seconds per action chunk. Turns out pip's default PyTorch for Jetson/aarch64 doesn't include CUDA, so every model was running on CPU with no warning. Pinning cu129 fixed it - cold start is now 1.44s and subsequent calls are basically instant. cu130 didn't work: it installs fine and torch.cuda.is_available() returns True, but crashes on the first actual transformer forward pass. cu132 is outside the allowed torch range for LeRobot 0.5.1. One thing to remember: don't trust torch.cuda.is_available() as proof CUDA works on this machine. Run an actual forward pass after any torch version change. Started recording SmolVLA datasets and running real inference sessions.
Week of July 21, 2026
First ROS 2 recordings; 3 cameras is too much for the USB bus
First recording sessions with the ROS 2 pipeline are working. Converting rosbags to a LeRobot dataset is about 4x faster than before since we encode video on the fly instead of writing PNGs to disk first. Hit a problem though: all 5 USB devices (2 arms + 3 cameras) share a single 480 Mbps USB 2.0 bus, and 3 cameras at once was too much. Kernel logs showed bandwidth errors. Staggering camera startup helped but wasn't enough so we settled on 2 cameras for now. There's an unused 10 Gbps USB 3 port on the Jetson that could handle a 3rd camera but it needs a self-powered hub. Also ran first SmolVLA inference and each action chunk took about 70 seconds. Something is clearly wrong, looking into it next week.
Week of July 14, 2026
Switched to ROS 2 Jazzy; teleoperation up and running
Moved to a full ROS 2 Jazzy workspace. First build broke right away because CMake picked up a uv-managed Python 3.14 that was ahead on PATH instead of the system Python. Fixed by pinning the Python executable in the build args - only needed once, CMake caches it. Set up udev rules so the arms and cameras get stable symlinks instead of /dev/videoN paths that shuffle around on reconnect. This was actually necessary from day one: a camera jumped from /dev/video0 to /dev/video2 after a USB reset. Teleoperation is running at 50Hz. Also figured out: if all 6 servos give zero response on a ping, it means the arm battery is off, not a USB or driver problem. The serial adapter shows up on USB regardless of whether the arm has power.
Week of July 12, 2026
TPU gripper works well on rigid objects; new member onboarding
Recorded new episodes with the rubber TPU gripper on the cube task. Noticeably better than the old grippers - the success rate went up and objects don't slip. Downside: it struggles with soft objects like fabric and sometimes flings things across the table instead of releasing properly. Still worth it for anything rigid. Also walked new members through the full pipeline from scratch - teleoperation, recording, training, and inference. DAgger is something we want to try at some point.
Week of July 5, 2026
Bimanual inference struggles; batteries swelled from overnight charging
Added a side camera to the bimanual setup for depth and ran first inference trials. Transfer task: the left arm had trouble reaching low positions, probably a depth issue. Pick-and-stack: the right arm barely moved, most likely because the cameras couldn't see it properly. A few models managed to pick up the block sometimes. Put the single-arm setup back for now so people can keep training and running models in the meantime. Reconfigured the bimanual rig so the depth camera sees both arms equally. Also: the 1500 mAh 11.1V batteries swelled up from being left on charge overnight. Don't leave batteries charging unattended.
Week of June 23, 2026
Bimanual setup done, starting pick-and-stack recording
Bimanual setup is complete and we started recording for a pick-and-stack task. Plan is to run ACT and try diffusion policy, though diffusion is probably going to be tricky. Need to figure out which parameters work best. On the hardware side: want to get a parallel gripper and explore new camera positions - either a twin-tower setup or 2 wrist cams and 1 overhead.
Week of June 16, 2026
Board wiping scrapped; Jetpack 7.2 fixes LeRobot dependency issues
Board wiping is a dead end. L1 loss barely moves the arm, L2 does more but still can't finish the task. Just too hard for now. Flashed Jetpack 7.2 which brought Python to 3.12 - LeRobot's dependency complaints are gone and PyTorch installs much faster. Had a bunch of camera issues too. Next up: record a stacking dataset (100 episodes, keep everything in frame), then move to bimanual.
Week of June 9, 2026
Red block picking is working well
Red block picking is in a good place. Best results came from checkpoints at steps 500, 1500, and 4000 with a temporal ensembling coefficient of 0.1. Going slow makes a big difference. Still have some dead zones we can't fully explain. Next: try the board wiping task, test quantized MolmoACT2, look into fine-tuning.
June 5, 2026
Cart setup is working, TPU pinchers grip much better
New cart setup is up and working. The TPU pinchers on the claw are a real improvement - objects don't slip during the task anymore. Should make training on rigid object tasks a lot more reliable.
May 30, 2026
Success rate jumps from 20% to 50% with one parameter change
Changed the temporal ensembling coefficient to 0.01 and the success rate went from about 20% to 50% on the block grab task. Motion is smoother too, less overcorrection between action chunks. Probably the biggest improvement we've gotten from tweaking a single parameter.
May 29, 2026
Cart setup in progress, temporal averaging experiments coming
Building a new cart-mounted setup with better camera positioning. Need to nail down a recording protocol for it. Going to try different temporal averaging values next session to see if we can improve smoothness and success rate.
May 25, 2026
150-episode model: moves too fast, claw lifts before it has a grip
Tested the policy from 150 episodes. Two problems: it moves too fast and the claw goes back up before it actually has the block. Next recording session we'll go slower and spend more time holding the object before lifting.
May 22, 2026
150 episodes recorded, ACT training done
Got 150 episodes of the blue block task recorded. ACT training is done (act_red_blue_lr1e-05). Trying inference today - if it looks decent we'll start logging success rates more formally.
Week of March 29, 2026
Training on the Jetson at 1000 steps is way too slow
ACT training directly on the Jetson is not practical right now. Even 1000 steps takes forever. Need to find better on-device training conditions and get a third camera set up to improve dataset diversity.
Week of March 22, 2026
ACT works on the tape boundary task
ACT is trained and working on the tape-boundary task - the arm places a block within the taped area on its own. Next: get the third camera set up and start looking into SmolVLA.
Week of March 15, 2026
Jetson setup done, first training run, dataset diversity is the problem
Moved teleoperation and inference to the Jetson (Jetpack 6.2). LeRobot commands work natively. Set up two cameras: side and grip. Recorded 100 episodes of placing a block within tape boundaries.
First training run had really erratic inference. The model overfits badly - any small change from what it saw in training and it freezes. One model just learned to hold the wrist roll motor at a constant value. Hardware and calibration are fine. The problem is the dataset. Need more varied starting positions and broader training sweeps.