chipx.cam

Key takeaways

• A model can only be used for its intended purpose, in a defined context, without taking unknown risks. 
• Models must be created using a well-defined process and verified in a way that provides a level of independence. 
• Deployment requires trust and a way to track the properties of the model.

A model captures some kind of behavior exhibited in the real world, but all models are compromises. They sacrifice some detail for execution performance.   The most important model arguably is the design, followed by a collection of models that form the verification process. But it also could be a model of a tool or a flow. AI is transforming the creation and utilization of these models.  Surrogate models of the design are often created to increase execution speed, leaving out some detail, or being approximate in nature. In many cases, the models are constructed hierarchically. An RTL model, for example, is a clock-accurate model of the design that utilizes gate or arithmetic-level models between clocked edges, which are themselves idealized from transistor models that are simplified models of physics. Each abstraction works so long as it is not used out of the context for which it was designed. Such models are used in both the analog and digital spaces. [An in-depth discussion of these can be found in ‘Can AI Create Missing Models?‘]  Model creation is generally expensive, in part because it needs to be verified. Models with unknown accuracy or fidelity, or used in a context outside of the original intent, can lead to bad decisions or missed problems. Both of these issues can be very expensive to correct if found later in the flow, or worse, if they escape into a product. If AI starts creating models, trust must precede adoption, and that only happens when a repeatable and traceable process is used. Even then, tagging is essential for attaching metadata to models to define their scope and suitability.  Examples of tool and flow modeling include DSO.ai from Synopsys, and Cerebrus from Cadence. These create optimization loops for block-based physical implementation tools based on reinforcement learning. After each trial, the PPA can be measured, and that new datapoint is used to improve the AI model. This influences a more informed choice of tool settings for the given design.  The primary input for these implementation optimization tools is the RTL design model. At that stage in the flow, it is fairly stable. Many of the AI models being created today are tackling areas of the design flow where things are not so stable, such as verification. Here, the point is to find errors in either the design or the testbench, and to correct them as quickly as possible while converging on a coverage goal for a target that may also be moving. Understanding the correctness and lifetime of the data being used to create the models becomes a very important part of the process.  Given that verification is a convergence process, it does provide some level of forgiveness in terms of data validity. “You can only really measure accuracy on in-domain data or ground truth simulations, which are expensive, so generalization is a challenge,” says Arvind Srinivasan, product engineering lead for Normal Computing. “If you train a very small model as a proxy for something else through imitation learning, as soon as the original representation changes structurally, you are somewhere else. There are two alternatives. First, learn physical rules that generalize, in which case the complexity is too high, and the only way you can do that is to over-reduce to something under-parameterized and over-broad. Second, remove the explicit physical constraints and try to learn via semantic understanding and use world models or LLMs to transfer knowledge.”  Tagging of analog behavioral models is equally important so that the model is not used in places where aspects of a model are required but not verified.  Creating models for design and verification  Many processes have been developed for model creation, and they will certainly change in the AI era. For purposes of this discussion, a fairly classical one is used. Mehir Arora, head of engineering at ChipAgents proposes a six-step process: 

• Define the model’s scope, interface, accuracy requirements and use context;  
• Generate reference data via golden simulations; 
• Produce the behavioral model; 
• Validate against the golden model; 
• Refine through iteration, and 
• Deploy into the simulation environment. 

Knowing how and when the model is going to be used is important, not only for model accuracy, but also ensuring that you minimize the work required to make the model. “You could have a very bespoke model that only your tool can use, or you can produce a model in the form of a SPICE network that gets wider deployment,” says Tom Demuer, research and development fellow at Keysight EDA. “There are very different effort levels between delivering a complete Verilog model, or just picking the simplest one that can perform the job.”  There are two important axes that have to be considered — accuracy and completeness. “They are different, and both matter,” says Ashish Darbari, CEO at Axiomise. “In digital verification, accuracy means comparing the model’s recommendations or predictions with outcomes we already trust. Completeness is harder because it is tied to what part of the verification problem the model is helping us cover. A model may look good on common scenarios, yet still miss rare protocol interactions, reset behavior, low-power modes, or security-sensitive paths.”  In the analog world, things are slightly different. “Accuracy is a statistical question,” says Hanna Yip, product manager at Normal Computing. “How close are the model’s outputs to ground truth across some distribution? Completeness is a coverage question. Does the model have behavior defined for all the scenarios that matter? You can have a highly accurate model that is deeply incomplete, one that produces outputs close to ground truth for everything it covers, but which has no behavior for important scenarios. Those two failure modes require different remedies.”  Knowing the usage context can help define the attributes of a model. “Correct use begins with defining the role of the model,” says Axiomise’s Darbari. “In digital verification, many of these are advisory flow models, not signoff engines. They can recommend regressions, prioritize tests, suggest assertions, or guide proof strategy, but they should not be allowed to declare closure on their own. That means putting them inside a bounded decision loop where simulation, emulation, formal results, or other trusted analyses remain the final authority.”  The creation of a model from data is similar to the problem of creating a model from a specification. “I want to give a spec to my large language model, and I want to have it generate a model,” says Paul Graykowski, director of product marketing at Cadence. “How do you guide the LLM in the right direction to generate code that’s valid, especially given that most specs are not very good? It doesn’t matter what you feed it — specs, timing diagrams — they all form part of a mental model, and there are a lot of things that must be explored that a spec maybe takes for granted.”  Creating a golden model  Assuming that the required model is going to be created from samples, simulations of a trusted model are used. “The quality of the training distribution matters as much as the volume of training data,” says Normal’s Yip. “For DRAM verification, the failure modes you care about, such as timing violations, protocol corner cases, or conditional interactions, are systematically underrepresented in typical simulation traces because most simulation time is spent on nominal scenarios. A model trained on that data will be confidently accurate on common cases, and silently wrong on the cases that cause re-spins. A deliberate sampling strategy in the data is as important as the fitting method, and it requires domain knowledge to get right.”  Analog creation is more physics-facing. “There, engineers run SPICE or EM across selected corners and stimuli, then train a surrogate or reduced-order model and package it as Verilog-A, real-number modeling, or another behavioral form,” says Darbari. “In both cases, the principle is the same: learn from trusted reference data and integrate the result back into the model.”  This flow may be suitable for a single model, but things become more complicated when a collection of models is involved, as would exist within a verification flow. Multiple data sources, multiple tools and multiple models are being continuously improved. “These models are usually created from data the verification flow already produces, including regression histories, coverage databases, debug outcomes, waveform traces, property results, and tool logs,” says Darbari. “That data is cleaned, labeled, and used to train ranking, classification, or recommendation models that can predict which tests matter, which failures are related, or where proof convergence is most likely. In formal, the inputs often include assertions, protocol specifications, abstract models, and prior proof results, so the AI is learning not just from simulation history but also from verification intent.”  From the reference data, a model can be created. “The agents analyze it to extract key parameters for the behavioral model,” says ChipAgents’ Arora. “Then, we iteratively produce a Verilog-A model that captures a subset of the behavior reflected in the reference data.”  None of the generation tools available in the market today is perfect. “Accuracy measurement for AI-generated design content requires distinguishing three properties that are not equivalent — syntactic correctness, functional correctness, and formal completeness,” says Simon Davidmann, AI and EDA researcher at the University of Southampton. “Treating these as interchangeable is one of the most consequential evaluation errors a team can make.”  We can expect varying degrees of correctness on this. “I can have an LLM write C or Python, and it’s going to be pretty accurate and run out of the box,” says Cadence’s Graykowski. “But if I am talking about SystemVerilog assertions or really complex UVM, it’s going to be hit or miss. And even though they’re all written to the SystemVerilog standard, the standard is kind of loose. It’s not perfect. The tools will give their best attempt at it.”  Others point to similar gaps. “Syntactic correctness, meaning whether the generated code compiles without errors, is a necessary but insufficient measure,” said Davidmann. “Current LLMs achieve near-perfect syntactic correctness on standard Verilog constructs. Functional correctness, measured by pass@k on benchmark suites such as VerilogEval, represents the fraction of generated modules that pass functional simulation within k attempts. However, these benchmarks are dominated by short, pedagogical-style modules that do not represent industrial SoC subsystem complexity. A model scoring 90% on VerilogEval may never have been trained on a cache controller with non-trivial CDC handling. The benchmarks in use today were designed for a world where humans wrote RTL. The field needs benchmarks designed to find out where AI breaks, and those benchmarks do not yet exist at industrial scale.”  There are also important techniques that can increase the chances of creating a good model. “If you can reduce to very compact representations, you are more likely to generalize because you probably learned something more fundamental,” says Thomas Ahle, head of machine learning at Normal. “Google recently released a strong weather model where they used very small bottleneck layers forcing the model to find a compact way of representing things, trying to discover the real physics of the system rather than learning a hundred million rules of thumb.”  Validating against a golden model  A common mistake is to use the same examples for verification that were used for training. “Accuracy can be verified by creating and running sanity test cases where we compare the generated stats against expected values and check if they match,” says Deepak Shankar, founder of Mirabilis Design. “Generated models need to be manually validated to make sure the AI models don’t hallucinate as the model becomes big.”  Sanity tests are important. “If you have S parameter data that you’re trying to model for some linear system — let’s say a filter — you will typically do a training set, but keep a holdout set so that you can validate whether your model matches the holdout set,” says Keysight’s Demuer. “Another important test is making sure that the surrogate model behaves physically. For instance, if you have a passive structure like a filter, the first thing to validate would be that your model keeps being passive. That’s a good litmus test. Similarly, your model needs to remain causal. You cannot traverse backwards in time.”  It is also important to recognize that, by definition, the models are not accurate outside of their intended purpose. “In behavioral models, waveforms do not necessarily need to match perfectly,” says Arora. “The model is meant to be a simplification of the true circuit behavior. Thus, the agent spends much of this phase reasoning about whether the deviations from the golden simulation are justified by the constraints of the model.”  You have to judge fitness for purpose. “On the analog side, behavioral and surrogate models simplify circuit behavior by design, so they can miss subtle effects or behavior outside the training envelope,” says Darbari. “These models may be powerful accelerators, but they do not remove the need for strong specifications, engineering judgment, and independent signoff evidence.”  There are various ways to help identify holes in coverage. “Along with the model, we generate plots that demonstrate the generated model’s behavior in comparison to the ground truth,” says Sidharth Kannan, agent research engineer at ChipAgents. “Human engineers are then required to sign off on the validity of the model before the agent is permitted to terminate. Every model and its outputs should be traceable and reproducible.”  When inaccuracies, errors, or problems are found, iteration is required. “This is where agents can dramatically reduce the burden,” says ChipAgents’ Arora. “They can perform automated gap detection where agents compare model against golden, identify where errors are highest, and automatically produce new tests to understand behavior in those regions. They can conduct active learning. Instead of running a predefined grid of simulations, agents can intelligently select the next simulation point that maximizes information gain. Then, once new simulation data is collected, agents can recreate the model and re-evaluate — without engineer intervention.”  Alternative flows attempt to incrementally build the model over time. This is especially true when modeling tools and flows, because it is not possible to build up a complete model ahead of time. That means refinement and deployment become tightly linked. “They learn from historical data, so they are often better at finding more of what the team has already seen than at exposing genuinely novel bugs or corner cases,” says Darbari. “They can improve coverage growth, but coverage is not the same as correctness, and strong-looking metrics can still hide important escapes.”  Simulation is essential 
Before deployment, there needs to be some kind of sign-off. “Before adopting any AI model into a production flow, the right question is not only how accurate the model is, but what it was trained on, what that distribution does not cover, and how close the actual design is to that boundary,” says Southampton’s Davidmann. “Every team deploying these tools should run a deliberate boundary-probing exercise on a representative sample of their most complex and novel design patterns, verifying the outputs independently against what a domain expert would produce. That exercise identifies where the distribution ends and the model’s uncertainty begins. Without it, deployment is proceeding without the information needed to manage the risk.”  It is important to understand the properties of the generated model. “Usually, it starts by tagging the model with properties,” says Keysight’s Demuer. “For instance, it’s a guaranteed passive model, it’s a guaranteed causal model. The next one might be where it is applicable — the parameter range under which you can trust results to be fair. After that, there is not much maturity in how to label a model in terms of accuracy, or the balance between accuracy and speed. What I’ve seen is that most teams have an underlying assumption that the model can be used and has been developed by a team that gives them good results.”  It can get complicated when there are restrictions on the data used to create the model. “It is important to put a watermark on any AI-generated content,” says Sathishkumar Balasubramanian, head of products at Siemens EDA. “Google just came up with a standard for any content that is AI-generated so that people know that it’s coming from an LLM. Having good data labeling is going to be very important. When you’re feeding data into your fine-tuned model, you have to make sure that you’re able to authenticate proper data. You can’t just say, ‘Go do whatever you want. Grab any data you want.’ You have to have a process to validate the data before you start fine-tuning it. It requires discipline, and you’ve got to do it all the way from the source.”  That implies trust. “I would not trust them 100%,” adds Demure. “Even in workflows where they are critical, they will typically be used in a hybrid form, where you’re using the AI-generated models. But you do, for instance, 50 evaluations of the AI model, and one with the true ground truth, so that you can validate over time whether you’re still within what you think you should have.”  Without trust, adoption will be limited. “One of the key properties of AI models in EDA is that reliability is bounded by training data distribution,” says Davidmann. “This constraint is well understood in machine learning theory, but rarely documented in vendor tool datasheets, and it has direct consequences for how these tools should be evaluated and deployed. Most teams do not yet have a standard for documenting which assertions are human-authored and which are AI-generated, or what that distinction means for sign-off confidence. In addition, most vendor documentation does not specify the training scope, failure modes, or domain boundaries of the models being deployed. Both gaps need to close before AI-generated content scales into safety-critical designs.”  The post An AI Model Fit For Purpose appeared first on Semiconductor Engineering.

Source: https://semiengineering.com/an-ai-model ... r-purpose/