I think OP didn't question the privacy of their Private Cloud Compute, just Apple's bad faith: they claim they can't handoff data in a privacy-preserving way to 3rd-parties when they tout that they absolutely CAN handoff data in a privacy-preserving way to their servers.
Apple frames this as a privacy issue when it's only a brand/control issue.
That’s right, they can hand it over to their servers. And they’ve got special agreements with Google to do the same exact thing. That preserves privacy.
Is it possible to do that with absolutely any company that wants to be able to be the AI on your phone? Are most of those companies even capable of handling something like that?
That’s thorny.
PCC is supposed to work only on Apple silicon. You are supposed to trust that the input will be decrypted within the enclave which is next to inference engine on the same box. This way you know the input does not leave the server. If they offload to another server (eg google) then the privacy boundary is broken, once it leaves the enclave. Microsoft does it differently, where inference is confidential so more guarantees if that could be replicated.
Google has a similar thing they announced a year or two ago that uses the various hardware security stuff in the PC world that Apple is working with them to add to the list of approved stuff that gives PCC level security.
Yeah I overestimated the PID stripping that was being done on-device before being handed off to a server. After other comments I realize there needs to be a lot of plumbing on the server-side too.