UniDex-ViTac: Learning Unified Visuo-Tactile Dexterous Manipulation Policy from Human Video Data
Human videos provide demonstrations of dexterous manipulation but lack robot-executable actions and tactile measurements. We present UniDex-ViTac, a framework that uses human-video-guided simulation to generate robot demonstrations paired with fingertip contact observations for training a deployable visuo-tactile policy. Object-specific residual reinforcement learning specialists adapt annotated human–object interaction references to a robotic arm–hand system. Their successful rollouts pair final robot action targets with robot-side fingertip contact observations. From 50 human demonstrations across ten objects, we collect 10,000 simulated trajectories to train a single Action Chunking with Transformers (ACT) based generalist. The policy combines point clouds, proprioception, and four binary contact signals encoded through fingertip labels and a separate token, without requiring human references or privileged object identity and pose at deployment. The contact-augmented configuration achieves 68.3% macro-average success in simulation, compared with 55.5% for the point-cloud-only baseline. Without real-robot demonstrations or policy fine-tuning, it succeeds in 73/110 physical trials (66.4%) across six seen and five unseen objects, compared with 60/110 (54.5%) for the baseline, an increase of 11.8 percentage points. These results support the feasibility of learning a unified visuo-tactile dexterous manipulation policy from video-guided simulated interactions.
Our pipeline reconstructs HOI trajectories from human videos, trains per-object Residual RL specialists to produce physically feasible robot motions, then distills 10,000 successful rollouts into a single multi-task Visuo-Tactile Generalist policy.
Each point in the 512-point cloud carries a one-hot label: (1,0,0) camera points, (0,1,0) non-contacting fingertips, (0,0,1) contacting fingertips. This is combined with a 42-D proprioceptive state and a 4-D binary contact signal, enabling the policy to jointly reason over scene geometry and local contact events.
Built on ACT with a CVAE encoder, the policy Transformer takes four tokens — point cloud (PointNet++), proprioception, binary contact, and a style latent — and predicts a 30-step action chunk of wrist pose and fingertip targets, executed via DLS arm IK and analytical hand IK.
(a) Our visuo-tactile policy achieves 68.3% average success on 10 YCB objects, outperforming ACT (53.4%), Diffusion Policy (45.2%), and BC-Transformer (20.2%). (b) Point cloud and binary contact are complementary — fusing both outperforms either alone (PCD Only: 55.5%, Contact Only: 58.4%). (c) Our dual fusion (onehot label + contact token) consistently performs best across all objects.
Deployed on a Franka Emika Panda with a 16-DoF dexterous hand, our PC + Tactile policy achieves 66.4% overall success across seen and unseen objects, outperforming the PC Only baseline (54.5%).
| Category | Object | PC Only | PC + Tac |
|---|---|---|---|
| Seen | bleach_cleanser | 10/10 | 9/10 |
mustard_bottle | 7/10 | 7/10 | |
power_drill | 7/10 | 8/10 | |
potted_meat_can | 4/10 | 7/10 | |
tomato_soup_can | 3/10 | 5/10 | |
pudding_box | 2/10 | 6/10 | |
| Average | 55.0% | 70.0% | |
| Unseen | maxwell_coffee_can |
9/10 | 9/10 |
brown_salt_box | 2/10 | 4/10 | |
plastic_wine_cup | 7/10 | 6/10 | |
blue_mug | 0/10 | 2/10 | |
pringles_bottle | 9/10 | 10/10 | |
| Average | 54.0% | 62.0% | |
| Overall Average | 54.5% | 66.4% | |
Table 1: Real-world success rates (out of 10 trials) comparing the Point Cloud Only policy and the Point Cloud + Binary Tactile policy.
@inproceedings{unidex-vitac2026,
title = {UniDex-ViTac: Learning Unified Visuo-Tactile Dexterous Manipulation Policy from Human Video Data},
author = {Anonymous},
note = {Under review},
year = {2026}
}