Featured2025Computer-vision and ML engineer

Robotic Vision for Clothing Detection & Segmentation

Built a containerized vision service that streams camera frames from a mobile robot and returns clothing detections or segmentation geometry.

GitHub

TL;DR

A client-server computer-vision system that keeps YOLOv11m and SAM3 inference off constrained robot hardware, serving bounding boxes, centroids, areas, and polygons through FastAPI HTTP and WebSocket endpoints.

Problem

A mobile robot needs to identify garments and locate their pixels precisely, but its camera hardware may be too constrained to run large vision models locally.

TL;DR

I built a containerized inference service that lets a lightweight camera device use YOLOv11m detection and SAM3 segmentation without carrying either model. The client sends one JPEG frame over a WebSocket, waits for its reply, and receives compact geometry that robot logic can use for navigation or manipulation.

Why it matters

A tidying robot needs to know both what a garment is and which pixels belong to it. Bounding boxes work for recognition, but their centers can land on bare floor between sleeves; segmentation masks provide a more trustworthy centroid for depth sampling and actuation.

Dataset / inputs

The detector was trained on Fashionpedia with 45,623 training images and 1,158 validation images across 46 clothing classes. Live clients send JPEG-encoded camera frames. YOLO returns class, confidence, and bounding boxes; SAM3 accepts text prompts and returns pixel masks that the service converts into geometry.

Technical decisions

Only one frame is allowed in flight per client. This applies backpressure when inference is slower than capture and bounds delay to a single round trip. Because SAM3 is too slow for every CPU frame, the service tracks masks with dense optical flow and re-segments periodically, after a large scene change, or when mask area drift indicates tracking failure.

Challenges

The public project is an intentionally limited demo of an earlier private system. Weights are external, standard Azure Container Apps profiles are CPU-only, and the research code for monocular depth, visual odometry, and occupancy mapping is not yet wired into the API. The next meaningful integration is projecting a mask centroid through calibrated depth and pose into a 3D target for the robot.

Methods

Fine-tuned YOLOv11m on Fashionpedia for garment detection
Served image, video, webcam, and WebSocket inference through FastAPI
Used SAM3 masks to compute centroids from image moments and return simplified polygons
Tracked masks with dense optical flow between expensive SAM3 passes
Enforced one in-flight frame per client to prevent unbounded latency on slow inference hosts

Results

Implemented HTTP endpoints for image and video prediction plus WebSockets for live detection and segmentation
Reduced full-resolution masks to robot-ready boxes, centroids, areas, and outline polygons before transmission
Isolated deployable service code in a CPU-oriented Docker image while keeping client and research code separate

Lessons learned

Backpressure is a robotics feature: bounding the queue prevents a moving robot from acting on stale frames
A mask centroid is safer than a box center when downstream logic samples depth on irregular objects

Limitations

Model weights are not distributed with the public repository and must be supplied or retrained
SAM3 is not real-time on CPU; optical-flow tracking reduces how often it runs but does not remove that cost
The monocular-depth and mapping research code is not connected to the inference API
The final 2D-mask-to-3D-world-coordinate step is not implemented

Next steps

Connect mask centroids to calibrated depth and camera pose for world-coordinate targets
Publish final mAP50 and mAP50–95 values from the Fashionpedia training run
Benchmark end-to-end latency on the target robot and deployment hardware

Related projects

Featured2025

Gap Junction Connectomics

Built a CNN-based pipeline that segments gap junctions in 3D electron-microscopy volumes and converts them into electrical-connectivity measurements.

Outcome: Created a reusable path from raw EM slices to 3D gap-junction predictions, per-neuron measurements, contactomes, and normalized electrical-connectivity matrices.