I've had success with this! It required a little bit of an existing network. I always wanted to be in a band but never could, I was never invited. But I went to an open mic for a couple months, and just decided to .. start one. Invited people over to jam sometimes. Turned into a regular event, then turned into a band.
I've repeated this a couple times. Yeah, usually I have to do the bulk of the inviting and organizing. And yeah, it's uncomfortable being the "leader". But I know everyone enjoyed the time together. Those that didn't just never came and that's fine too.
You really can just do things!