When you dig through the CUDA developer docs instead of the promotional materials, you can develop a view of Nvidia GPUs as having 8-128 processing cores, each with 4 hyperthreads, running 32-lane SIMD for almost everything. Where a lane is 32 bits wide.

The promotional material likes to label the individual lanes as “cores” because it sounds more impressive. And, it’s not entirely incorrect.

Even the dev docs use the marketing terminology. The description I gave above needs a bit of piecing together.

SPIR-V states its an int, float, vector n (where n <= 4) or a matrix (2..4 cols of vector n).

It does not necessarily mean the hardware can do 4x4x64 floating point operations in a single subgroup operation, but at least the programming model supports framing it that way.