Deploying visual defect detection in manufacturing confronts three challenges familiar across image processing domains from medical imaging to semiconductor fabrication: extreme scarcity of real defective training data, the need to generalize across product variants without retraining, and regulatory demands for explainable decisions. This talk presents our industry experience addressing these challenges through a two-stage cascade framework combining supervised anomaly detection trained on synthetic-only data with vision-language model (VLM) reasoning, and the practical lessons learnt from deployment in production.
Stage 1 of our framework employs CLIP ViT-L-14 with Defect-Aware Prompt Optimization, trained entirely on synthetic data, to generate defect probabilities and class activation heatmaps under 200ms. A confidence gate routes ambiguous cases to Stage 2, where an 8B-parameter VLM reasons over a visual triangle: the test image, the Stage 1 heatmap, and a optional-good reference image.
The key enabler for real-world generalization is domain knowledge injection at inference time. Manufacturing engineers author a structured defect taxonomy which is semantic description for each defect type with visual identification criteria (e.g., “the rectangular hole behind the hexbolt is partially visible when the washer is missing”). Embedded in the VLM prompt, this taxonomy grounds reasoning in domain-specific cues and constrains outputs to valid categories, requiring zero model retraining for new product lines. The VLM’s pre-trained visual representations simultaneously bridge the synthetic to-real domain gap: when Stage 1 predictions degrade on real production images, the VLM compensates by reasoning over actual pixel content guided by the injected defect taxonomy. This enables deployment with zero real defective training samples across three product families spanning 3–8 defect classes.
We discuss deployment tradeoffs from production experience, including confidence threshold tuning, the cost-accuracy balance of the two-stage gate, and why natural-language reasoning traces proved essential for operator trust in regulated environments.
