It does affect your ability to speak, and not necessarily only negatively.

You lose more evolved vocabulary and formal grammatical structures, but you gain a liveliness and a colloquial dexterity that bookish people often lack.

You can also be more inventive in your metaphors and figures of speech, instead of relying on the usual stock cliches - something often seen in oral interviews with illiterate folk.