The "city" field in the mailing address is less city and more post office name. USPS is a lot more flexible now, but used to be, if your city didn't have its own post office, your mail would need to be addressed to the city where your post office resided; and even if your city did have a post office, it doesn't mean that's the one that services your delivery point.
Many people don't even live in an incorporated city, so at a municipal level they're just in a state and a county, but not a city. Those people still have to put something in the address field, but that leads to a lot of confusion. Unincorporated towns often have no official basis, and aren't always unique either. Of course, neither are incorporated cities in some states. :P
Changing ZIP codes to be geographical would significantly reduce the usefulness of ZIP codes for their primary use, sorting and delivering mail, so that's a non-starter. Use something else!