For about $2000, I picked up a Mac Studio with 96 gigs of RAM, which is effectively all VRAM thanks to Apple’s architecture. It doesn’t have the raw number-crunching power of the big GPUs but with all that space, you don’t need to worry much about the size of the model until they start getting really big, so it’s pretty easy and flexible.
I’m able to do basically everything that others are doing with AI, entirely locally. It generates text and writes code (it’s no Opus, but probably on par with the best of a year and a half ago), images, videos, songs, all that stuff (those last few are garbage but it’s basically the same level garbage that the cloud models are making). And I have total privacy, will never get a surprise price hike or lose access to a model I like, and know exactly how many bottles of water I’m using to cool it (zero).
It’s not heroin, it’s weed. There will be a market for it, but, like, you can also just make it yourself. 90% of what people want will just get done on device. I can’t see any way this turns into a trillion dollar industry.
Apple proposed something a few years ago, when governments were making similar threats, that attempted to strike a middle ground. The idea was that upon uploading an image to iCloud Photos, a on-device scan would be run on that image and an encrypted report generated to be sent up along with the photo. There was differential privacy involved, the report would also sometimes be generated for entirely normal photos, so seeing a report didn’t necessarily indicate anything, and they had set it up such that the server would only be able to decrypt the reports if it had a sufficiently large number of photos that had been actually found to be CSAM by the local scan, so there would have to be many false positives to incorrectly get flagged.
It was incredibly controversial, and they ended up not doing it after all. In my opinion, it’s probably the lightest touch and most responsible way to do something like this, and obviously they always pick the most worthy cause for invading privacy… but I still viscerally dislike the idea that my computer would have code designed to try to get me sent to prison under certain circumstances (not that I’d ever be triggering that code with anything but a false positive of course). Somehow it’s worse than just saying “in the cloud you have no privacy, your photos aren’t encrypted on our servers, and if you upload CSAM we’ll drop a train on you.”