Clearly our math system gives basic math, so we should conclude that it is either 'incomplete' or broken. Before the incompleteness theorem, it was exciting to imagine that it might be complete, but we had no idea. Now it seems likely to be incomplete. The odds of it being broken haven't really changed.
If it is broken, 2+2 still equals 4 (we didn't really learn that fact from abstract math), but we'll need to fix up set theory. In a world where math isn't formally verified (and even lean itself isn't formally verified), the Incompleteness Theorem is permission to be awed by how huge the space of all logical deduction is, not an argument that logic is a fiction.
To come at it another way, do you believe that there is an 64-bit integer which, any time it is inserted into register 8 on any computer, will immediate convert the raw mass of the earth into a black hole? No, because even if you suspect there is a hole in the foundation of math, you still believe in brute-force case-analysis, and you know that every 64-bit integer has been inserted into register 8 on a computer before.