Generative engineering and the role of simulation

For the past few months, the Artificial Intelligence (AI) community has been buzzing about Generative AI tools that can create arbitrary content, such as ChatGPT or Midjourney. According to a recent study by McKinsey, Generative AI’s impact on productivity could add trillions of dollars in value to the global  economy. In this blog, we’ll delve deeper into what Generative AI and generative engineering truly entails, how it can be utilized in engineering, and the role of simulation in the process. In the field of Product R&D alone (excluding software engineering), the study of McKinsey predicts a potential market size of $300 billion.

The Age of Generative Algorithms

Generative AI is a subset of AI technologies (c.f., Figure 1). It can produce content such as text, images, sounds, animations, or even 3D models, depending on the instructions provided. It achieves this by learning the patterns and structures from a large amount of input training data, enabling it to generate content with similar characteristics.

Most of these solutions rely on various deep learning approaches, with unsupervised or semi-supervised learning methods being dominant for training. What’s fascinating is its ability to learn in diverse ways and from vast amounts of data that lack labels. Many of the foundational deep learning technologies, such as variational autoencoders (VAE), generative adversarial networks (GAN), diffusion models, and transformer networks, have been in existence for some time. However, only recently we have gained access to sufficient data (for instance, Midjourney is trained on over 100 million images) and the necessary hardware (for instance, ChatGPT-4 has been trained for 100 days on 25,000 Nvidia A100) to train Generative AI models that have billions of parameters.

This progress has enabled the development of exceptionally large and potent foundational models, such as GPT-3/4 and Stable Diffusion. These models serve as the backbone for creating AI systems capable of handling multiple tasks. For example, ChatGPT, built on the GPT-3/4 framework, can draft essays from brief text snippets, while Stable Diffusion can render highly realistic images based on textual descriptions.

Generative Al in Engineering – What is (and isn’t) Generative Al?

With the remarkable successes of Generative AI across various disciplines, questions about its application in engineering naturally arise. But let’s start by putting Generative AI into an industrial perspective: There are already two well established terms: CAD (Computer Aided Design) and CAE (Computer Aided Engineering).  ‘Aided’ being the operative word. Computational algorithms implemented in software applications that aid you in your engineering design endeavors.  However, just as a pencil is an aid in writing a novel, you still must write the novel yourself. The word ‘generative’ implies creative ability. Instead of doing it yourself (despite any aid in doing so), it is generated on your behalf.

Historical examples of basic AI involve image recognition. Provide an image as input and have the AI return some classification of it, e.g., this is a dog, or cat, or mop, or muffin, to varying levels of difficulty:

Generative AI on the other hand can directly ‘generate’ an image for you from some verbose natural language prompt (taking an engineering example):

“Design a sketch of a sportive convertible with off-road-tires“

Thereby, Generative AI accommodates intent. An incorporation of multiple requirements, interpreted from the same language that you would otherwise instruct a human intelligence to generate. This example showcases the creative potential of AI. It assumes the role of a designer or engineer, crafting a unique car concept autonomously. In subsequent discussions, we’ll use the terms Generative Engineering or Generative Design, which are the industry’s most recognized nomenclatures for this process.

This virtual design use cases is one of the major use cases in engineering, according to a recent McKinsey Study. Further major use cases in engineering are early research analysis, virtual simulations, and physical test planning. Within this blog we will focus on virtual design addressing Generative AI to generate drafts and designs, allowing engineers to iterate more quickly and to better explore all design freedom. In earlier blogs, we have investigated virtual simulations accelerated by new deep learning techniques (How Machine Learning is pioneering change in engineering) as well as the opportunities of Generative AI offering  unprecedented user experiences in terms of copilots (The Potential Impact of LLMs on CAE).

But before delving further into the nuances of Generative Design or Engineering, it’s essential to understand the particularities of engineering in the context of Generative AI. Namely, how to get the data for training corresponding models and how to assess the results in the context of engineering.

Generative Al in Engineering – How to get the data?

To do so, let’s examine the limitations of the above example. DALL·E2 is trained on a vast collection of images sourced from various image repositories. These repositories encompass a wide range of objects, image styles, and so forth. However, while in general many images as well as other data sources are freely available (c.f., Figure 4), it is challenging to find such depth and richness of data for engineering cases. Engineering data often contains significant intellectual property and therefore isn’t shared across different industrial companies. Imagine the DALL·E2 demo above, but with a Generative AI trained exclusively on pictures from a specific car company. The results would be considerably restricted. But the world of engineering is changing. For example, in a recent survey 80% of the respondents would share their data to train an AI tool, if they would be allowed to run their CFD simulations for free. Furthermore, technologies like few-shot learning or transfer learning show promise to either use less data or benefit from data across domains. Though they will likely need more time to achieve industrial maturity.

Generative Al in Engineering – How to assess the results?

Another significant distinction between Generative AI for producing content like text, images, sounds, animations, and its application in engineering lies in validation and testing. The standard cycle in the use of Generative AI incorporates a human in the loop, assessing and validating the quality of the outcomes and iterating — since the ideal solution is seldom reached on the first attempt — until a satisfactory result is achieved.

In typical engineering scenarios, merely having a human in the loop is insufficient. Engineering heavily depends on the evaluation of outcomes through (virtual) testing. Thus, any Generative Engineering process must incorporate virtual testing within the Generative AI loop (c.f., Figure 5). It’s imperative for this process to comprehend the foundational physics, that is, the cause-and-effect relationships. Consequently, simulation emerges as a pivotal tool in the application of generative technologies within engineering.

Therefore, the swift advancements in the realm of Generative AI will not only revolutionize how we approach engineering but will also introduce new demands on simulation technology. This dynamic will be a primary catalyst for innovation in the field of simulation. As we discussed in our previous blog post titled Beyond Traditional Engineering Simulations, AI is also set to play a significant role in this transformation.

Generative Al in Engineering – Where to start?

While generating content like an image is relatively straightforward (an image consists of a well-defined and structured set number of pixels), the process of engineering (typically covering quite diverse data types) presents a substantially more complex challenge. According to a recent review of Yann LeCun (A Path Towards Autonomous Machine Intelligence, 2022) a key challenge towards more Generative AI is the planning of complex action sequences, as also typically found in engineering tasks. This typically involves learning hierarchical representations of action plans. Therefore, let’s take a hierarchical perspective on engineering first. Starting from a Model-based Systems Engineering point of view, which is the de facto engineering process in many industries, will help us to better asses opportunities of Generative AI in engineering.

Model-based Systems Engineering (MBSE) is the formalized application of modeling to support system requirements, design, analysis, verification, and validation activities beginning in the conceptual design phase and continuing throughout development and later life cycle phases- (Definition according to INCOSE).

The V-model (c.f., Figure 6) provides an abstraction of the concepts, workflows, and ideas behind MBSE.

For many of the sub-steps in the V-model, Generative AI presents a unique opportunity to elevate the automation of complex system design. In the following discussion, we’ll adopt a high-level perspective. We won’t delve into the translation of requirements into formal models, such as SysML v2, where generative LLMs can play a significant role, but will instead concentrate on the creative design process itself. However, it’s worth noting that the conceptualization and formalization of system engineering workflows are crucial elements in enabling Generative AI workflows in engineering.

Generative Engineering

Every design process begins with the creation of a functional architecture and system-level design. Traditionally, this has been an expert-centric engineering process. However, AI technologies provide a unique opportunity to automate this architectural search. At Simcenter, we have been pioneering related ideas for several years. Simcenter Studio (c.f., Figure 7) generates and evaluates system architectures during the early concept phase. It combines system simulation, optimal control methods, and reinforcement learning with a cutting-edge machine learning and scientific computing stack to simulate and assess hundreds of architectures based on the requirements. The result is a selection of architectural options from which the engineer can choose. In most use cases, it will provide a significant faster optimal system architecture compared to today’s heavy manual and expert-centric processes.

Figure 7: Generative Engineering with Simcenter Studio

Generative Design

Once a system architecture is defined, the detailed design phase begins. This might involve selecting the right components from a modular system or designing suitable new components. The latter often requires devising new 3D geometries, which is what is typically associated with the term design. The algorithmic creation of these new geometries is what we term Generative Design.

While the phrase Generative Design is relatively recent (having emerged around 2013/2014), the concept itself isn’t new and can be traced back several decades. Innovating new topologies, for instance, for lightweight design, as well as optimizing shapes, such as for drag reduction, have been areas of study for a long time. Many of the foundational technologies are based on a mix of mathematical optimization and traditional simulation methods. It’s only in recent years that machine learning algorithms have been incorporated into Generative Design tools.

Generative Design typically begins with a set of design criteria, like required strength and weight, and an initial design envelope shape or constraints on shape, such as the maximum space available. An iterative algorithm then generates design proposals. The process begins by analyzing a shape via simulation, identifying KPI sensitivities or gradients in relation to design variations, and subsequently pinpointing a more optimal shape using gradient-descent type algorithms. Augmented by additional ML-type algorithms, not just one design is proposed but a range, allowing the engineer to choose from. This facilitates finding an optimal balance between potentially conflicting requirements of performance KPIs and aesthetics. Examples of generative design tools include the NX Topology Optimization for Designers solution (c.f., Figure 8) or the Topology Optimization feature in Simcenter STAR-CCM+.

While technologies like Topology Optimization don’t necessitate vast data sets to create a new design, so-called surrogate modeling technologies can be employed to generate effective new designs based on previous ones. Using (variational) autoencoders, the core features of designs are learned, which can then be applied to explore similar designs. Such approaches work exceptionally well when the foundational geometries are parameterized.

Classical Design Optimization

The central concept of Generative Design is to have devised a new shape based on specific requirements. However, many engineering tasks are less intricate, such as optimal sizing tasks. In these instances, traditional mathematical optimization technologies, like those provided by Simcenter Heeds, serve as an excellent starting point for enhancing design efficiency and producing highly optimized, superior designs. Given all the fuzz around Generative AI, let’s not forget about classical methods, which are often a great starter as well as good fine tuners for many applications.

The Future of Generative AI in Engineering

While the examples above demonstrate that initial Generative AI solutions are ready for integration into today’s engineering workflows, there’s still a considerable journey ahead before they achieve the same ubiquity as they do in content creation, such as text, images, sound, or video. The primary challenges currently are the availability of data (which includes rethinking business models) and enhancing the autonomy of simulation. This enhancement would ultimately enable its use without human interaction where all functional and non-functional requirements are satisfied, directly. Even though promising algorithms, like transformers, exist today, we’ve only just begun to realize the potential of Generative AI to revolutionize the productivity of engineering workflows.  In our Technology Innovation team, we are researching various aspects around Generative AI in the context of Engineering including the implications on future simulation technology. Thus, stay tuned for future news.