GPU me up, Scotty!

By Stamatina Petropoulou

Spoiler alert! With Simcenter STAR-CCM+ 2022.1 we are opening the door to a new era of CFD simulation speed-up techniques. With NVIDIA GPU-enabled acceleration you can now achieve faster turnaround times running your CFD on GPUs – at a significantly lower per-simulation cost.

Space: the final frontier. These are the voyages of the starship Enterprise. Its continuing mission: to explore strange new worlds, to seek out new life and new civilizations, to boldly go where no one has gone before
https://en.wikipedia.org/wiki/Where_No_One_Has_Gone_Before

The Transporter – beaming me up

Let me try and show you what this means for you with a little story from my childhood. It was the time when streaming was not a thing. Serious anticipation for the next episode of your favourite TV series was the norm. Being a geek, I was a Star Trek enthusiast. I would wait for the whole week to sit in front of the TV. Me, my dad and sister fixated on the voyages of Captain Kirk, his team, and the strange new world they would explore next.

I should say, it was the whole experience that I was yearning for; the space travel, the mission, the discovery of new life and civilizations, pushing the boundaries of humanity itself. But what fascinated me the most was the Transporter. The crew would beam themselves to any place they wanted at lightning speed. Just like that, they would get on with their new adventure and fulfill their mission. It was the fact that transportation was taken to a new level of efficiency that fascinated me. As a child, I was not worried by the physics (or lack of) behind it. Or the whole debate of whether this was feasible or not. It served a purpose and made their endeavor possible. I wouldn’t have it any other way!

Beaming you up, all a dream!

Much like Transporting in Star Trek, most engineers use CFD simulations as a necessary part of their job. Only this time they do care about the Physics behind it! But the main interest is their mission; to design a better car, an airplane or even an oven. CFD is just the transporting system to get them there. It helps them to complete their mission! So, in those terms, CFD should be as efficient as possible. As such, faster turn-around times at lower computational cost and hardware investments is an ongoing demand. A demand that exists in any company leveraging CFD simulation technology for efficient product development.

CPU vs. GPU – the race has just begun

The evolution of CPUs is seemingly slowing down in terms of Peak Memory bandwidth and Peak Double Precision. At the same time, GPUs are definitely on the rise. Quoting Dr. Vincent Natoli , “When compared on a chip-to-chip basis against CPUs, GPUs have significantly better capability on both speed of calculation (FLOPS) and speed of data movement (bandwidth) (GB/s)”. The figure below tells this story

Simcenter STAR-CCM+ strives to maximize the use of available compute power and NVIDIA GPUs are an increasing proportion of this. But why did we choose to enable GPU acceleration now? The rapid development in GPU architecture in the last years, especially with NVIDIA introducing High Bandwidth Memory cards, allowed for better collaboration between the CPU and the GPU. Additionally, the introduction of more flexible programming languages allows larger parts of the code to be ported to GPUs faster.

Now is a new era. Now is the time to GPU you up! With CFD on GPUs

This large increase in bandwidth made with GPUs brought significant speedups in unstructured meshes. It made it prime time for CFD calculation on GPUs with Simcenter STAR-CCM+.

Another benefit of using GPUs for CFD is their inherent massively parallel computation capabilities. GPUs are known to deliver significant scalability and better price for performance than CPUs. All this at a lower total cost of ownership.

The all-new Simcenter STAR-CCM+ 2022.1 GPU-enabled acceleration provides end-to-end physics solver functionality on the GPU (for the supported solvers). This is critical to providing exceptional GPU performance and utilization. In this first release, we are leveraging the NVIDIA CUDA platform and the AmgX solver. Simcenter STAR-CCM+ 2022.1 will include single and multiple GPU support on Linux operating systems. On the physics side, it supports both steady and unsteady, constant density flows using the segregated solver. GPU-based calculations are compatible with most turbulence models, including RANS, DDES and Reynolds Stress Models. Additionally, GPU enabled acceleration supports most standard reports, monitors and field functions. This makes sure no overhead occurs from using this functionality.

Accelerating our existing solvers and framework with GPUs, provides our users a seamless experience from CPU to GPU. Any case constant density case, already setup with the segregated solver, can run on GPU and return the same solution faster and cheaper.

On the hardware side, NVIDIA GPUs with Volta architecture or newer are supported.

Cards with HighBandwidth Memory (HBM2) are preferred over Graphics Double Data Rate 6 (GDDR6)
NVIDIA® V100 or NVIDIA A100 Tensor Core GPUs in HPC systems, and NVIDIA RTX™ A6000 in workstations

Benchmarking vehicle aerodynamics CFD on GPUs

We made the test, leveraging GPUs in various industrial vehicle aerodynamics benchmarks. The benchmark proves, CFD engineers can reduce hardware cost to 40% of the CPU equivalent and the power consumption down to 10% of the CPU equivalent. All this while maintaining identical simulation turn-around times. That equates to 8x NVIDIA V100 Tensor Core GPUs giving the same time to solution as 25 to 29 dual-socket Intel Cascade Lake Xeon Gold nodes with 40 cores per node.

CPU hardware – in terms of the equivalent number of dual-socket Xeon Gold nodes – required for the same time to solution as achieved with 8x NVIDIA V100 Tensor Core GPUs. Benchmark cases: Corvette C6 ZR1 external aerodynamics, plus three other industrial vehicle aerodynamics cases with resolution of 80-110m cells.

Then we took one of the benchmark cases to a scalability test. Using 1 to 3 NVIDIA 8x V100 32GB proves that scaling across GPU nodes still gives benefit, but not linear. More importantly, engineers can achieve a significant further speed-up by using NVIDIA’s latest GPU technology: One single NVIDIA 8x A100 80GB DGX achieves an equivalent runtime to 2120 CPU cores!

Comparison of GPU- and CPU-based equivalent simulations: GPU run time and number of CPU cores required to reach *almost identical turn-around times for the* same vehicle aerodynamics CFD on GPUs and CPUs

More bang for the buck with CFD on GPUs on the cloud

If we translate those numbers into the cost of running on the cloud we realize that on NVIDIA GPUs we get a significant cost reduction. An instance of GPUs 8x NVIDIA V100 costs approximately $25, 8x NVIDIA A100 are about $33, while an instance of CPU (1 dual-socket Xeon Gold node) costs $2.10 dollars.

Comparison of estimated costs to run the same aerodynamics case in almost identical run time running CFD on GPUs or CPUs

In a quick calculation from our benchmark, we estimate the NVIDIA A 100 GPUs to be more than 3 times less expensive, or in other words, enabling compute cost savings of up to 70%.

“Siemens Simcenter STAR-CCM+ is giving an incredible boost to CFD simulations by using NVIDIA GPU technology via the CUDA platform and accelerated libraries. Simcenter STAR-CCM+ users can now run more simulations, faster, and can gain critical insights for their design and operation workflows without compromising on accuracy leveraging NVIDIA GPU architecture.”
Niveditha Krishnamoorthy, Developer Relations Manager at NVIDIA

The voyages of the Starship Simcenter STAR-CCM+ with NVIDIA GPUs has only just begun

With this first version of NVIDIA GPU-enabled acceleration, Simcenter STAR-CCM+ will enable aerodynamics CFD engineers to massively improve their simulation throughput at equivalent compute investments. But for the development team of Simcenter STAR-CCM+, this is – while significant – just a first milestone. We plan to extend GPU-enabled acceleration across all relevant core solvers bringing GPU-enabled acceleration to CFD simulation engineers across all industries and applications.