Large Language Models (LLMs) have revolutionized the way we work with text, generate code, and even understand visual content. But their application is no longer limited to software or content creation.
In the EdgeAI project, we are exploring how multi-modal LLMs—models that understand both images and text—can accelerate and improve the hardware development workflow.
Why use LLMs for hardware? Hardware development is often slowed down by the need to manually interpret complex documents such as: circuit timing diagrams, datasheets and handwritten notes etc. With LLMs like LLaVA (see Figure 1), which combine computer vision and natural language processing, we can automate much of this work—turning unstructured visual data into machine-readable formats and enabling interactive processing and analysis.
Figure 1. Architecture of using LLM to handle image
Here are two examples from EdgeAI project and the research.
Case Study 1: automating production report processing
Production reports are used in manufacturing and quality control to log issues, perform root cause analysis, and document resolutions. These reports often come in scanned images or PDFs, making them hard to process programmatically. We used (1) LLaVA to read image-based production reports; (2) extract fields such as “Problem Description”, “Root Cause”, and “Corrective Action”; (3) convert the visual content into structured machine-readable format.
This can save engineers hours of manual data entry and enables cross-project traceability and analytics at scale.
Case Study 2: interpreting complex timing diagrams
Timing diagrams describe the behavior of signals in digital circuits. Interpreting them requires precision and context awareness—skills that even experienced engineers sometimes find challenging. We extended the use of multi-modal LLMs to (1) parse and label waveform signals from timing diagram images; (2) detect signal relationships (e.g., setup/hold violations); (3) answer natural-language queries like:
“Does this SPI timing sequence violate any constraint?”
This opens the door to AI-aided debugging, where engineers can interact with diagrams via chat-style prompts.
Why It Matters? AI isn’t just a tool, it’s becoming an intelligent collaborator in engineering. Their impacts involve (1) reducing manual effort in documentation and debugging; (2) improving collaboration between AI and hardware engineers; (3) unlocking valuable data buried in images and PDFs.
These work are not just accelerating hardware engineering—also changing how we think and build.
*(Figure 1 is drawn by PhD student from TUWIEN.)
Blog signed by: TUWIEN team