Waterloo Visuomotor Research

Abstract

This work studies the complete visuomotor imitation-learning loop in a simulation-first research platform. A two-finger gripper must pick up a randomly placed cube, move it to a target box, release it, and lift clear. Although the task is compact, it includes contact-rich manipulation, visual and geometric randomization, explicit termination rules, and the timing constraints of closed-loop deployment.

The system connects human teleoperation, a ground-truth scripted expert, successful policy rollouts, synchronized episode recording, behavior-cloning training, asynchronous inference, and deterministic checkpoint evaluation. The current policy combines frozen DINOv3 visual features with action-history and pose tokens, then predicts a future action horizon. At runtime, observation, replanning, and execution run on separate clocks so neural inference never blocks the 100 Hz controller.

System Overview

The project is organized around one shared data and timing contract, from the first demonstration to final closed-loop evaluation.

1 Collect Teleop, scripted expert, or policy rollouts

2 Record Frames, actions, poses, and provenance at 100 Hz

3 Train Behavior cloning over whole-episode windows

4 Deploy Asynchronous action-horizon replanning

5 Evaluate Seeded rollouts and checkpoint comparisons

Task

Pick a cube from a randomized table location, place it inside a target box, and retract the gripper within 30 seconds.

Variation

Cube pose, target-box position, cube dimensions and mass, colors, lighting, camera viewpoint, and movement speed.

Success

The cube remains inside the box while the gripper lifts clear. Timeout or an unreachable cube terminates the attempt.

Data Collection and Control

The same interactive simulator supports demonstration collection, model playback, hybrid intervention, and detailed diagnostics.

Human teleoperation. Mouse and keyboard commands provide direct demonstrations and targeted interventions.

Three sources, one episode format

Episodes can come from human teleoperation, a scripted expert using ground-truth MuJoCo state, or successful inference rollouts. Hybrid episodes preserve a model/teleop label at every timestamp.

Frames: synchronized overview images
Actions: command deltas and target residuals
State: mocap, palm, cube, and finger poses
Metadata: task, result, source, seed, and randomization

A background worker writes complete HDF5 episodes and optional MP4 review videos without pausing simulation. Failed automated attempts are discarded by default.

Asynchronous inference with three clocks

The control loop separates observation sampling, neural replanning, and trajectory execution. Single-slot queues discard stale requests and results, so the controller always works with the freshest available information.

Observe10 Hz
Capture the training-aligned observation window and recent 100 Hz action history.
Replan5 Hz
Run the policy on a background thread and build a time-stamped trajectory.
Execute100 Hz
Interpolate the latest trajectory and apply guarded Cartesian and grip commands.

Policy Architecture

The current model uses a compact multimodal tokenizer rather than training a large visual backbone end to end.

RGB framesObservation window

Action history[dx, dy, dz, grip]

Pose stateMocap, palm, cube, fingers

Frozen visual encoderDINOv3 ViT-S+/16

Patch features pooled to 24 visual tokens

Spatial fusionToken Transformer

Type, position, and temporal embeddings

Temporal contextSequence Transformer

Learned-query pooled observations

PredictionAction Horizon

[dx, dy, dz, grip] × N

Now-anchored actions

Future Cartesian targets are represented relative to the current observation, keeping training and live control aligned.

Action chunking

The model predicts multiple future controls at once, providing a smooth trajectory between slower neural replans.

Frozen feature cache

Precomputed float16 DINO tokens remove repeated encoder work and provide approximately 9× training throughput.

Architectures explored

Family	Architecture	Research question
Baseline	State MLP	How far can privileged low-dimensional state go without images?
Visual	CNN + MLP	Can a spatial-softmax CNN support fast direct action regression?
Visual	DINO ConvNeXt	How do frozen convolutional DINOv3 features compare with ViT tokens?
Sequence	CNN Transformer	Does a transformer action decoder improve temporal prediction?
Generative	Diffusion head	Can conditional denoising model multimodal action sequences?
Generative	Flow matching	Can continuous flow prediction provide a simpler generative head?
Ablation	Camera-only + auxiliary head	Can explicit localization supervision replace privileged cube position?

Checkpoint Evaluation

We measure how reliably the learned policy completes the task, showing how success and completion speed improve throughout training.

Six benchmark charts showing success rate approaching 99 percent and average completion duration decreasing to approximately three seconds across checkpoints — Representative checkpoint sweep. Success rises toward 99%, failures approach zero, and successful completion time falls to approximately three seconds.

99%best evaluated success rate

Seededidentical episodes per checkpoint

Completeaggregate and per-episode reports

PortableCSV, HTML, video, and W&B summaries

Qualitative Results and Diagnostics

The visualization tools expose the policy’s planned motion and align spatial trajectories with every recorded control signal.

Action-horizon preview. Colored markers show successive predictions and the active time-stamped trajectory.

Episode inspection in Rerun. Video, 3D paths, commands, poses, and residuals share one timeline.

SO-101 hardware teleoperation. Early hardware-control work toward a simulation-to-real deployment path.