That's a pretty good point. We kept hearing these rumors for a while now and all the usual suspects keep saying "we don't listen to your microphone on your phone" but yeah nobody expects the TV and fridge to do it, so maybe that's how it happens and people are not crazy.
That's just weasel language. For "hey, Google" to work, the phone needs to listen. Just like it can react to "hey, Google", it can also react to any amount of other trigger words without ever actually recording your conversation.