This is exactly what we’ve been doing.
For example, a client want a classifier to decide to make a go/no go decision on roofing jobs based on weather. The end result doesn’t use LLMs at all, but does use a lot of other techniques including human feedback when it makes the wrong call.
(Now the work crew gets the decision at 5 AM instead of 5:30 and the guy making the decision doesn’t have to wake up at all.)