Say the TV finds such an access point and uses it. It phones home and so now the manufacturer now knows that such and such serial # TV is located in that neighborhood.
They still need to tie it to you. If you've not given it a shred of personal information, it has to be fingerprinting via surreptitious audio/video recording, or else by obtaining the purchase info from the retailer (like the name and billing address from your credit card) and matching that by serial number (which would have to be recorded during purchase).
I wouldn't be surprised if they did audio fingerprinting. But even if they don't, the third parties who are buying the data can easily re-identify the users by doing data fusion with other sources (smartphone location data from other vendors)