Training is compute bound.

If its memory bandwidth bound you need to raise your batch sizes to leverage more parallelism.