The frustrating part about an LLM is at a fundamental level it's designed to do the wrong thing, it's designed to provide an answer for things it can't do, e.g. one argument I've been in recently at work is someone using it to summarise numerical data. They've a raw data spreadsheet they're putting into ChatGPT and asking it to produce a report in a particular style. So the LLM is doing its job, predicting the tokens to emit, and not actually doing mathematical analysis of the underlying data.

I had a team proudly show me how they were feeding their dashboard data into an LLM to predict growth needs, instead of doing basic linear regression or any number of pre-existing basic statistical analysis. The part that depressed me the most on that was it was a bunch of engineers that did it, the people you'd most hope would understand what an LLM is/does.

This is exactly what we’ve been doing.

For example, a client want a classifier to decide to make a go/no go decision on roofing jobs based on weather. The end result doesn’t use LLMs at all, but does use a lot of other techniques including human feedback when it makes the wrong call.

(Now the work crew gets the decision at 5 AM instead of 5:30 and the guy making the decision doesn’t have to wake up at all.)