I think we hit the ceiling with transformer -architecture long time ago. It is questionable how much sense there is on model training. I’d prefer we would put our effort in creating more efficient hardware and better software applications using these models.