Phan Binh
As an Undergraduate Student at The University of Da Nang - University of Science and Technology (DUT), I have developed and optimized multimodal deep learning and computer vision models, including attention-based architectures, for applications such as product categorization and manure identification. These models were designed to achieve high accuracy and efficiency, effectively handling imbalanced data and performing well in resource-constrained environments, ensuring robust and real-time performance.
I specialize in real-time change detection, object detection, single-camera tracking, multi-camera tracking, generative AI, and crowd estimation. My work focuses on deploying advanced AI models on embedded platforms and optimizing them for real-world applications.
Research
My research focuses on developing and optimizing multimodal deep learning and computer vision models, including attention-based architectures, for real-world applications such as product categorization, manure identification, and AI-powered assistive technologies. I specialize in creating solutions that achieve high accuracy and efficiency while effectively handling imbalanced data and performing well in resource-constrained environments.
My expertise spans various technical frameworks including Python, PyTorch, TensorFlow, ONNX, OpenCV, and ROS2. I have extensive experience in data preparation, model optimization techniques (quantization, pruning), and diverse learning methodologies including transfer learning, supervised, self-supervised, and unsupervised learning. My practical work includes real-time change detection, object detection, image classification, image segmentation, crowd counting, generative AI, and single/multi-camera tracking, with a strong emphasis on deploying these models on embedded devices like NVIDIA Jetson Orin NX, Intel NUC, Odroid N2+, and Arduino microcontrollers.
If you are interested in Computer Vision, Deep Learning, and Embedded Systems, feel free to contact me for collaboration or mentorship in these areas.
Work Experience
PV Module Layout Optimization & Rooftop Detection System:
• Designed a 2D bin-packing optimizer for PV module layouts on single- to four-slope roofs, honoring code setbacks, obstructions, azimuth/tilt, and spacing constraints to maximize energy yield and roof utilization.
• Reconstructed roof geometry from plan dimensions (CAD/PDF) using Gemini Vision API to extract building dimensions and roof gable overhang orientation from Japanese architectural drawings, with multi-page PDF processing.
• Built a computer-vision pipeline to detect and vectorize rooftop facets from aerial imagery: YOLO11x-seg for roof polygon detection, ridge line detection using Hough Transform, and roof type classification (1-4 slopes) with DBSCAN clustering and RDP simplification.
• Tech: Python, OpenCV, NumPy, Gemini Vision API, PDF2Image; Models: YOLO11x-seg; Methods: Dynamic Programming, Knapsack algorithm, CLAHE preprocessing, adaptive thresholding, DBSCAN, RDP simplification, Hough Transform, multi-threaded processing.
Automated Document Processing System for Japanese Real Estate Using Vision-Language Models:
• Developed an automated document processing system for Japanese real estate using Vision-Language Models (QwenVL) to extract and populate data from PDFs into Excel templates.
• Implemented end-to-end pipeline for document digitization: detecting empty cells in templates, extracting structured data from PDF documents, mapping extracted information to predefined schemas, and generating formatted output files.
• Built REST API with FastAPI, asynchronous processing, callback support, and error handling.
• Integrated LLM for table detection, OCR, and form filling with prompt engineering.
• Results: automated document processing workflow, reduced processing time from manual to automated, high accuracy with Japanese text and complex tables.
• Technologies: Python, FastAPI, QwenVL, Vision-Language Models, OCR, PDF Processing, Excel Automation, RESTful APIs, Docker, Pydantic.
AI-powered smart glasses for the visually impaired:
• Integrated object detection, depth estimation, text to speech, speech to text, and vision-language models on embedded platforms like NVIDIA Jetson Orin NX, Intel NUC, and Aria smart glasses to assist visually impaired users. Achieved real-time performance up to 14 FPS on Jetson Orin NX.
• Enabled real-time assistance for visually impaired users by guiding them to locate specific objects upon request and navigate safely by detecting and avoiding obstacles along their path.
YOLOv5s Pruning Optimization: Pruning YOLOv5s by 10% reduced inference time by around 5%, with a minor mAP drop (from 0.812 to 0.797). At 50% pruning, inference time decreased further, but mAP dropped to 0.654. Performance declined significantly beyond 50% pruning. Tests were conducted on an Odroid N2+.
YOLOv5s Transfer Learning: Enhanced the YOLOv5s model using Transfer Learning with pre-trained VOC weights, improving object detection on a smaller dataset. The mAP increased from 0.695 to 0.812, then further to 0.846, demonstrating significant accuracy gains with limited data. A slight increase in inference time on the Odroid N2+ (up by 0.88%) was deemed negligible compared to the accuracy improvement.
Quantization YOLOv5s: Applied Quantization techniques to compress the YOLOv5s model, utilizing both Post-training Quantization and Quantization Aware Training. Post-training Quantization reduced inference time by 3.57% with a slight mAP drop (from 0.824 to 0.806). Quantization Aware Training further enhanced speed (reduced by 9.45% compared to the original model), maintaining mAP at 0.819 with minimal accuracy loss. These results, tested on the Odroid N2+, achieved faster inference times with high accuracy retention.
Counting people in the ROI area of many CCTVs:
• Locating and Analyzing Areas of Interest: Users can seamlessly select and analyze specific areas within the video stream by clicking on optional points that define the corners of the Region of Interest (ROI).
• Dynamic People Detection and Counting: The system leverages the YOLOv8n model to detect and count individuals within the ROI, delivering real-time updates with approximately 98% accuracy, a processing time of about 40ms per frame, and a performance of around 25 fps on an NVIDIA GTX 1650 (4GB).
• Advanced Analytics: By integrating cutting-edge technologies such as OpenCV and Streamlit, the platform delivers robust data visualization and real-time analytical capabilities.
Enhanced Attention-based Multimodal Deep Learning for Product Categorization on E-commerce Platform:
• Model Design and Implementation: Designed and implemented an attention-based multimodal deep learning model to improve the accuracy of product classification on e-commerce platforms. The model integrates image and text data using a robust fusion module with attention mechanisms to classify 16 product categories.
• Performance and Impact: The proposed model achieved a significant accuracy of 91.18%, outperforming traditional multimodal and unimodal deep learning models (which reached a maximum of 77.21%). This improvement enhances product searchability and the overall customer experience.
• Personal Role: My role involved building and optimizing the deep learning architecture, focusing on fusing multimodal data for better product categorization. This model contributes to solving the challenge of automatic product classification, with practical implications for e-commerce platform management.
The modified Vision Transformer for imbalanced NIRs data classification:
• Model Development: Led the design and implementation of NIRsViT, a deep learning model based on the Vision Transformer architecture. This model was specifically tailored for manure classification using near-infrared spectroscopy (NIRS) data to improve identification accuracy across various manure types.
• Innovative Methods and Performance: Introduced innovative methods such as Focal Loss and Upsampling to handle imbalanced datasets common in agricultural data. This approach significantly enhanced classification performance, achieving an F1-Score of 93.03% and an accuracy of 97.96%, outperforming existing models.
• Responsibilities and Contribution: My responsibilities encompassed developing the deep learning model, preprocessing data, and optimizing classification algorithms. This project established a new benchmark in NIRS-based manure identification, laying a foundation for future research.