Real-time visual tracking for a 2-DOF pan-tilt camera rig using Model Predictive Control (MPC), Kalman state estimation, and deep learning vision wrappers.
The goal of this project is to track targets by compensating for visual processing latency, frame drops, and motor acceleration limits in an eye-in-hand feedback loop.
The system controls a 2-DOF Pan-Tilt platform driven by NEMA stepper motors, using visual feedback from a camera mounted on the tilt axis.
-
Vision Pipeline: Detection interface (YOLOv8, MediaPipe, OpenCV) mapping target centroids to normalized angular deviations
$(\Delta \theta_{\text{pan}}, \Delta \theta_{\text{tilt}})$ . - State-Space Model: Kinematic state model representing $[\theta_{\text{pan}}, \dot{\theta}{\text{pan}}, \theta{\text{tilt}}, \dot{\theta}_{\text{tilt}}]^T$.
- State Estimation (Kalman Filter): Kalman Filter for target state estimation, noise dampening, and forward prediction to compensate for visual processing lag.
-
Control (MPC): Online trajectory optimization formulated via
cvxpyrespecting acceleration limits, and motor actuation constraints. -
Hardware Bridge: Serial communication bridge (
SerialBridge) interface delivering step and direction commands from an ESP32 to motor drivers.
src/vision/- Detection pipelines, YOLOv8/MediaPipe wrappers, and target coordinate extraction.src/control/-cvxpyMPC solvers and Kalman filtersrc/hardware/- Serial communication bridge and microcontroller motor driver protocols.src/setups.py- Tracking scenario definitions and scenario configurations.
The pan-tilt rig was sketched and built from start to finish, by using appropriate items found in my garage. The electronics were soldered onto a protoboard, which was sketched beforehand. The sketches and final protoboard can be seen below.
- Microcontroller: ESP32
- Motor Driver: TMC2209 Stepper Driver
- Motor: NEMA17 Stepper Motor
Run the main visual tracking pipeline (runs a simulated version if ESP32 is not connected):
python main.py