We just want it to understand how to write code. We don’t also need it to know how to grow a potato.
I think perhaps you misunderstand how much of being an effective coder is understanding business domain enough to not be constantly asking for clarification (or if one is a fool or an ai, assuming wrong answers). I reckon a vast collection of trivia on the level of knowing how to grow a potato is important for a programmer
And you can't know ahead of time, when you're training the model, what business domains it will be used for. Someone may decide to use it to optimize the watering and fertilizer cycles of their automated potato-growing setup, and suddenly the "how to grow a potato" texts that went into training the model are actually the very things that make the difference between success and failure for the code the model spits out.
The disjoint set of English related to strictly growing potatoes and adding features to code is a lot smaller than you probably think...
It is hard to cut out a huge portion of English and truly understand English and human language.
You're just not saving as much as you might assume you could.
{"deleted":true,"id":48317190,"parent":48315485,"time":1780012523,"type":"comment"}
... unless the software is potato farm software.
Programming is not a rare skill, the interaction with domain knowledge is.
To me, the magic with LLMs has always been on the input side. It needs to understand what you mean in order to do what you ask. Most people are pretty terrible at communication, and general world knowledge seems to help with that.