That's because Panpsychism is silly.
I don't think it's any more silly than the alternatives.
You don't think it's sillier than "rocks don't have consciousness"?
Potential for consciousness. To have a working mind you need to have a way to gather, process and store information. In case of rocks it would be very empty, static, timeless experience.
If I believe that machines or neural networks can develop consciousness, then I also believe in some kind of substrate independence which presupposes some kind of panpyschism imho because as you say things would have a potential for consciousness, some more and some less.