Engineers often use vision-language models to produce new designs, such as for airplane or automobile components. To simulate how those components will perform in realistic situations, they’ll use tried-and-true computer-aided design (CAD) software to generate 3D models of those designs, which they can put through virtual crash or durability tests. Researchers from MIT and elsewhere have now developed a system that can teach a vision-language model to automatically convert 2D designs into CAD programs that are much more accurate and functional compared to other approaches, while using only a fraction of the computation. By improving the performance and efficiency of AI-driven CAD generation, this technique could streamline the rapid prototyping process and reduce costs. It could also help engineers identify beneficial design choices they might otherwise overlook. The……

Instead, the MIT researchers built a data augmentation system called GIFT (which stands for Geometric Inference Feedback Tuning) that generates data designed to improve the performance of one VLM for a specific task. GIFT develops an understanding of the model’s strengths and weaknesses by testing it. Then it uses this knowledge to generate data that could improve the model’s performance on the CAD generation problems it struggles to solve. “We want to obtain data augmentation that is informed by the model itself,” Giannone says. Learning from mistakes To do this, GIFT asks the model to generate code that solves a CAD generation problem multiple times in parallel.