SemiEngineering: The Murky World Of AI Benchmarks

Excerpt from article: “The Murky World Of AI Benchmarks”

AI startup companies have been emerging at breakneck speed for the past few years, all the while touting TOPS benchmark data. But what does it really mean and does a TOPS number apply across every application? Answer: It depends on a variety of factors.

Historically, every class of design has used some kind of standard benchmark for both product development and positioning. For example, SPECint, dhrystone, and geekbench are used as a set of benchmark programs for processor performance. RFC 2544 is used to benchmark networking ASICs. But the development of more customized and heterogeneous hardware, coupled with continuous updates in AI algorithms, has made decisions based on benchmarks murky at best.

“In recent years, we have seen a lot of advancements in hardware choices available for AI – from GPUs to FPGAs to custom hardware ASICs,” said Anoop Saha, market development manager at Siemens EDA. “However, the existing benchmarks are not suitable for measuring how the hardware will work for AI applications – either in training or in inference.”

Read the entire article on SemiEngineering originally published on April 30th, 2020.