Someone would likely get prosecuted if they instructed AI agent to run say a pump and dump scheme...

Even if the final output doesn't have copyright protection it might still be copyright violation. I think it could be reasonable to have work that itself violates copyright when distributed even if it does not have copy right itself.