Yes. Assumptions are being made.

It sounds like I should be considered a criminal because someone else might indeed be a criminal and I disagree.

If the next argument is you are unable to tell the difference myself and the criminal, then that is on you not me.