A complete computer vision and robotics integration project using ArUco markers, homography mapping, and the EEZYbotARM MK2 to autonomously detect, track, and sort colored objects.
This system combines advanced computer vision techniques with robotic control to create an autonomous object sorting solution. Using a downward-facing camera and ArUco marker-based homography, the system accurately maps pixel coordinates to real-world millimeter positions on a workspace. Objects are detected, tracked through multiple frames, classified by color and orientation, and then picked and sorted by a 4-DOF robotic arm.
The project demonstrates practical applications of OpenCV, inverse kinematics, serial communication, and state machine design for real-time robotic control.
Four corner markers establish precise coordinate mapping from camera pixels to workspace millimeters with sub-mm accuracy.
Sophisticated tracking algorithm maintains object identity across frames, handles occlusions, and manages appearance/disappearance.
HSV-based color detection with voting system ensures accurate classification (RED, GREEN, BLUE, YELLOW) even under varying lighting.
Calculates object angle using minimum area rectangles with circular mean smoothing for stable rotation measurements.
Serial communication with Arduino-controlled EEZYbotARM using inverse kinematics for precise pick-and-place operations.
Runs at 30+ FPS with state machine logic ensuring objects are stable before picking, preventing false detections.
PICK <id> <x> <y> <angle> <color> <target_x> <target_y>DONE, system removes object from tracking
# ArUco Marker Detection & Homography
detector = aruco.ArucoDetector(aruco_dict, params)
corners, ids = detector.detectMarkers(gray)
H, _ = cv2.findHomography(image_points, world_points)
# Object Detection (HSV-based)
hsv = cv2.cvtColor(frame, cv2.COLOR_BGR2HSV)
mask = cv2.inRange(s, 21, 255) & cv2.inRange(v, 0, 249)
contours = cv2.findContours(mask, cv2.RETR_EXTERNAL)
# Coordinate Transformation
world_pos = cv2.perspectiveTransform(pixel_pos, H)
// Inverse Kinematics Calculation
base = atan2(y, x) * 180 / PI;
distance = sqrt(reach² + z_adj²);
// Law of Cosines for Joint Angles
cosElbow = (L1² + L2² - d²) / (2 * L1 * L2);
elbowAngle = acos(cosElbow) * 180 / PI;
// Smooth Servo Movement
for (int pos = current; pos <= target; pos++) {
servo.write(pos);
delay(MOVE_DELAY);
}
# Install dependencies
pip install opencv-python opencv-contrib-python numpy pyserial
# Clone repository
git clone https://github.com/BenjaminKemmitz/vision-robot-sorting
cd vision-robot-sorting
# Upload firmware to Arduino
1. Open eezybot_controller.ino in Arduino IDE
2. Calibrate arm dimensions (ARM_SHOULDER_LENGTH, ARM_ELBOW_LENGTH)
3. Adjust servo limits (SHOULDER_MIN/MAX, ELBOW_MIN/MAX)
4. Upload to Arduino Uno
# Configure workspace dimensions and COM port
# Edit configuration in homography_robot_sorting.py:
XMAX = 663.0 # Workspace width (mm)
YMAX = 316.0 # Workspace height (mm)
ROBOT_PORT = "COM3" # Arduino serial port
# Run the system
python homography_robot_sorting.py
Print 4 markers (IDs 0-3) from DICT_4X4_50 dictionary. Place at table corners in clockwise order starting top-left.
Measure distances between markers in millimeters. Update XMAX and YMAX values in config.
Measure shoulder-to-elbow and elbow-to-gripper lengths. Critical for accurate inverse kinematics.
Define target positions for each color in SORT_ZONES dictionary. Ensure all zones are reachable by arm.
STATE_NEW → Object first detected, position tracking begins
STATE_STABLE → Position stable for 8 frames, color voting active
STATE_QUEUED → Ready to pick, waiting for robot availability
STATE_PICKING → Robot currently executing pick sequence
STATE_PICKED → Successfully sorted, removed from tracking
def circular_mean(angles_deg):
"""Handle 0°/180° wrapping for angle averaging"""
angles_rad = np.deg2rad(angles_deg)
sin_sum = np.mean(np.sin(angles_rad))
cos_sum = np.mean(np.cos(angles_rad))
mean = np.arctan2(sin_sum, cos_sum)
return np.rad2deg(mean) % 180
# Hue ranges for different colors (OpenCV uses 0-180)
if h < 10 or h > 170: return "RED"
elif 35 < h < 85: return "GREEN"
elif 100 < h < 130: return "BLUE"
elif 15 < h < 30: return "YELLOW"
30-35 FPS
Real-time vision processing on standard hardware
±2-3mm
Homography-based coordinate transformation precision
~15 sec/object
Complete pick-and-place cycle including stabilization
95%+
Successful picks under optimal lighting conditions
vision-robot-sorting/
│
├── homography_robot_sorting.py # Main vision system
├── eezybot_controller.ino # Arduino firmware
├── SETUP_GUIDE.md # Complete setup instructions
│
├── assets/
│ └── aruco_markers/ # Printable ArUco markers
│
├── docs/
│ ├── calibration.md # Calibration procedures
│ └── troubleshooting.md # Common issues & fixes
│
└── requirements.txt # Python dependencies
Automated part sorting, quality inspection, assembly line pick-and-place
Teaching computer vision, robotics, and control systems integration
Vision-guided manipulation, human-robot collaboration studies
Robotics competitions requiring autonomous object manipulation
Implemented circular mean to properly average angles near 0°/180° boundary, preventing jitter.
HSV color space with voting system provides robust classification across lighting conditions.
Distance-based matching with missed frame counter maintains tracking through brief occlusions.
Reachability checking prevents invalid arm commands, with graceful fallback for unreachable targets.