What "privilege" means in this context is that you need a license to do it. That doesn't imply that it isn't still a right. For example, you might need a signage license to put up a sign, but free speech is still a right, and for that reason the government is constrained in the criteria they can use to deny you the signage license.

But people say "driving is a privilege, not a right" as if these things are alternatives to each other. Requiring you to pass a driving test is quite a different thing than discriminating against you based on your state of residence.

Here's a quote from your own case:

> The nature of the private interest involved here (the granted license to operate a motor vehicle) is not so great as to require a departure from "the ordinary principle . . . that something less than an evidentiary hearing is sufficient prior to adverse administrative action," Eldridge, supra at 424 U. S. 343, particularly in light of statutory provisions for hardship and for holders of commercial licenses, who are those most likely to be affected by the deprival of driving privileges.

Strongly implies that constraints exist on what the government can deny. What's that about, if there isn't a right to be implicated?