I'm not sure LLMs will be good at doing taxes any time soon primarily because they're very bad at even simple math unless it's explained in words like in proofs. A good illustration of this is asking an LLM something like this:
"If I put a container on a food scale, tare it, add 100g of peanuts, tare it, weigh 100g of almonds and then remove the container from the scale, what does the scale read?"
LLMs completely puke all over themselves on this one. They do not account for not taring after the almonds and they do not see the weight of the container needs to be subtracted. The correct answer is minus (100g + container weight). Although this particular question also exploits another LLM weakness which is physical world concepts. LLM are just guessing. For an LLM to do taxes well, it would need to be trained on many scenarios and it would need a math service that it can use as a support service.
You just need to give an instruction something along the lines of "any initial calculation or derived value must be computed through the use of a $LANG program."
Pretty sure the LLMs could implement tax calculations better than the average accountant, were they given the rules.
In my experience agents will reach for numpy for any non trivial math without being instructed
As a veteran tax CPA, I think LLMs can be very good at preparing taxes but I would not trust them to complete a tax return start to finish. An LLM that can take data from tax documents and input that data into the correct boxes of tax preparation software has no need for math, it just has to identify data in a pdf, spreadsheet, or handwritten note and transcribe that data to the correct box in professional tax software that then does the math. In your typical local CPA firm, an LLM could replace the unlicensed preparers/interns/early career stage CPAs in filling out the return for the experienced professional to review and adjust according to their specialized knowledge. As with many industries, I think the LLMs are suited at replacing entry level automated turks but lack the skills necessary to fully complete the project.
Look at multiple posters here explaining that their accountants made O($10k) errors that were discovered by LLMs.
I’m not sure the evidence is on the side of the CPAs. Maybe CPAs with LLM assistance.
“The scale reads −(100 g + the weight of the empty container).”
I literally gave it your question. What models did you test?
ChatGPT and Cursor both failed hard. Even after walking them backwards through the flaws in their answers they just refused to see it. When I stated flat out what the answer was ChatGPT agreed in the end. Cursor did not. Cursors initial reply was -200g. Then I challenged "But I did not tare it after adding the almonds." and again it failed with: "That doesn't change the final reading: it still displays -200g.". When I walked it back further with "If I put the container on the food scale, tare it and remove the container, what does the scale read?" it correctly replied "negative the weight of the empty container" but ... when I asked "So does this impact the agents answer to my original question?" it replied "No, it doesn't change the answer—the scale still reads -200g". LLMs are NOT good at even basic numeric accounting.
You keep saying LLMs can’t do this but I get the correct result each time without drama.
Cursor uses Grok so far as I can tell, which is.. not great and I hope you did not try ChatGPT free or 5.6 or Luna.. you cannot unfortunately just say “Claude” or “ChatGPT”. Their lineup is sufficiently complex - each with their own failure modes - to warrant some further specification.
I think you are “directionally” correct, as these clankers like to say, because language models are indeed not known for their arithmetic wizardry. However, as my correct answers show, they can do it just fine.
Size, unfortunately, matters.
Free version of Gemini also gets it right
{"deleted":true,"id":50285725,"parent":50279446,"time":1793260429,"type":"comment"}
{"deleted":true,"id":50280639,"parent":50279446,"time":1793220321,"type":"comment"}
[dead]
[dead]
[dead]