Phone calls and SMS should be like that, which, they almost could be today just with the phone OS keeping track of who you've previously "accepted."[1]
Except.
The identifiers (phone numbers) are nearly infinite, and nearly free, so the scammers just use a "new to you" number every time they call you, allowing each one to generate a new 'invitation.'
And of course to make it worse, the "numbers" are actually truly free since they can spoof any number they want all day long and to this day, most of those calls will go through and not even show a big red flag.
That last part is entirely reprehensible that our carriers haven't solved it by now, but apparently they don't want to.
[1] and also there's that sticky problem that some arbitrary company like say, a health insurance company or the state government, has 1000 departments who might call you, and they couldn't give you a full list if they wanted to.
How often have you met people over the phone in the past decade?
I will say that blocking all unrecognized numbers, though, is an unworkable idea. Any parent or caregiver knows that they need to be able to occasionally receive calls from a number they've never heard of before.