I think you are correct for the overwhelming majority of cases for the US; The unauthorized reproduction/distribution is where things get very aggressive and easy to prosecute based on existing case law.

The only case that comes to mind as far as trying to threaten just for downloading, blew up in the law firm's faces... among other shenanigans, it came out their own machines were seeding files as an attempt to honeypot.

However other countries may have different laws as far as possession vs distribution and related penalties.