The 'Elbow method' [1], yeah. Sweep a bunch of different k values and pick the one where your target metric starts having diminishing returns. There are a lot of heuristics you could use.
There's still not one particularly great way to pick k for this type of clustering. There kind of can't be as clustering is sort of inherently subjective.
Other methods like HDBSCAN can find the number of clusters themselves. There's still a 'one key parameter' you have to pick yourself, but it can be more intuitive and less sensitive for some datasets, when you don't know how many clusters there are / how many you want.
Sometimes, picking k exactly is useful for practical reasons too. Like fixing the size of your codebook. Or partitioning a dataset into k equal-ish pieces, say to spread across a pool of servers of a known/fixed size.
Yeah, I guess you can't really avoid picking k without hierarchical clustering.
Yeah, it's not a well-defined problem, so there's no one true solution. Then only question left is "well what do you want?". Always have to pick something.