Yup. Took the cost of the network adapters back out. Thanks. Some days. . . .

The Gaudi3 system does not require installation of additional network cards or GPU linking, according to SMCI

https://youtu.be/BQVfGdkiTtc?t=120

https://www.intel.com/content/www/us/en/content-details/817486/intel-gaudi-3-ai-accelerator-white-paper.html

The LINPACK HPL benchmarks are used for the top500 ranking, so the FP64 matrix design decision apparently also held back Aurora on that overall ranking, but Aurora did take top spot on the LINPACK AI workloads.

https://www.intel.com/content/www/us/en/newsroom/news/intel-powered-aurora-supercomputer-breaks-exascale-barrier.html#gs.auil5e

“Aurora supercomputer secured the top spot in the high-performance LINPACK-mixed precision (HPL-MxP) benchmark – which best highlights the importance of AI workloads in HPC.”

Jay, here is a direct quote from Rick during the HPE Argonne Aurora briefing ahead of ISC, when Rick was asked about how come Aurora takes so much more energy on LINPACK than other machines:

“So let me answer that in probably the simplest way. So underlying hardware for Frontier and Aurora are different. So Frontier, the AMD 250X, has a has a matrix engine that executes 64 bit math which means that for matrix calculations that it can do basically twice the performance of the vector unit when it is doing 64 bits. Aurora does not have that matrix unit, and that was a deliberate design decision because most scientific calculations can’t take advantage of 64 bit dense matrix calculations. LINPACK can, but other scientific codes can’t. So the actual real applications, not the benchmark, do quite well on Aurora, we have a number of them that are multiples of the equivalent performance on Frontier. But the cost of doing that is you’re running the benchmark in vector mode. So that’s the underlying reason. But it was a deliberate design decision to not use silicon for a matrix unit for double precision, we put that extra silicon into accelerating lower precision on the PVC. So that’s — in BF16, for example, we have a lot more performance. So that’s, that’s that’s a technical reason. If you buy me a beer, I’ll tell you more about it.”

This was, as you might imagine, news to me. But that is what Rick said.

Rick Stevens was integrally involved in the specification of the PVC GPU explicitly for a mixture of HPC and AI processing. If he had wanted 64 bit matrix processing, it would have been included.

The Gaudi3 AI processors also do not include 64 bit matrix operations.

https://cdrdv2-public.intel.com/817486/gaudi-3-ai-accelerator-white-paper.pdf

Key bit of data missing. According to Rick Stevens, who runs Argonne, it doesn’t have 64-bit matrix math, just 64-bit on the vectors. Which is what I meant.

Ignorance is a strong word. Tired is a less strong one, which I am a lot these days for reasons I do not care to explain.