Hacker News new | past | comments | ask | show | jobs | submit
lexandstuff | on: Dayjob: Tax – evaluating AI agents on 50 professional tax assignments
I prepared my taxes this year using Astra + Codex. It did a thorough job on a complicated return, and I sent it to my accountant to fill out the tax form, with everything laid out in a spreadsheet. They sent it back for final review, and I had Codex check it; it was riddled with errors. The refund was nearly $2k lower than it should have been, and they put a bunch of stuff into the wrong fields. Had to go back and forth multiple times to get it 100% correct. Made me wonder how many previous years' returns were wrong.

I feel like we're going to get to a point where it's mandatory to use AI for certain tasks - humans just can't be trusted to get things right that require a level of precision.

idopmstuff | parent | next
I sent my business and personal returns through Claude this year before signing, and it turns out they were a horror show of unforgivably stupid mistakes. Pretty infuriating considering what my now former CPA was charging me.

He amortized the goodwill of businesses I acquired over 5 years instead of 15 (incredibly basic stuff, which Claude flagged with *CRITICAL ERROR*). When I asked about it, he said he had caught the error in his review but forgot to save after he fixed it. I don't know if there's a good response to making that kind of a mistake, but if there is, it's definitely not that one.

fn-mote | root | parent
> When I asked about it, he said he had caught the error in his review but forgot to save after he fixed it.

Obviously total bullshit.

Did you fire him?

Ask for damages?

loading story #50290989
loading story #50283050
e63f67dd-065b | parent | next
> Made me wonder how many previous years' returns were wrong

Feed previous year returns into Astra and find out! It's not too late for amended returns, you have 3? years from original date of filing

dgellow | parent | next
You all have zero concerns sending the AI providers your personal and business financial data?
aprilthird2021 | parent | next
Your anecdote seems to clash heavily with the results of the benchmark announced in the article. I suppose that is part of the jagged experience of agentic AI
mlyle | root | parent
The benchmark announced in the article seems to be around weird edge cases (which are often important) and whether it can prompt the person with relevant information rather than just listening to what they want to do.

Top tier tax professionals can do pretty well on this benchmark (better than agents) but won't hit 100%.

The kind of people spending time preparing oodles of returns would not do very well on this metric, though (especially midway through that pile of returns, as opposed to "think carefully about this trick question").

aprilthird2021 | root | parent
The OP said it was a complicated return so anyone assuming OPs anecdote is in line with the article is filling the lines in themselves with a lot of assumptions