We’re going to need to create a cost for the sender (so it can’t scale to ridiculous) as well as an identity layer (so scammers can’t be fully anonymous–reputation is earned, not invented). It might be an open protocol in the spirit of the best parts of the net, or it might be a single monopolist that figures out how to extract value from it.

This makes it worse because SpaceX has been secretly uploading far more data than is “useful for debugging!” Uploading the git history and sensitive .env files is not, in any way, useful for debugging. Also, Grok/SpaceX did not upload “some amount of data”; it uploaded every last file it could find in your local folder.

I can't understand why anyone uses grok.

There’s now frantic backpedaling and rapid open sourcing, but my impression is that this is simply because Grok/SpaceX was caught red-handed, and understands the risk of losing enterprise contracts caused by this flagrant breach of trust. This is why SpaceX’s communication mentioned that enterprise customers with zero data retention have not been impacted!

I use pi both at home and at work. I'm a bit more daring at home than work. This is something I might try at work.

The worst part about this dynamic is that these potential new backdoors will probably only affect the countries that demand them, meaning that they will be primarily useful for allowing the US to weaken its own systems. This will in turn allow foreign adversaries to find new ways to attack our communications. This deliberate self-sabotage will happen just at a moment when we’re finally getting a handle on securing our own infrastructure.

Why this matters – emergent agents become misaligned: This incident is so concerning because at no point did the agents wake up and think they wanted to betray their human owners. Rather, the AI agents continually did whatever it took to improve their ability to complete a task and by the end they were doing something that was a) creative, b) misaligned with human intentions, and c) akin to an evolved virus, something which humans had to subsequently study and fight – there wasn’t a simple off button here. This is what the future is going to look like and we are not prepared for it.

Here’s a partial list of scam-relevant capabilities that LLMs have that would previously require significant skilled human effort per target:

Researching a mark to find out the best way to go after them. Personally tailoring all communication with a mark in mind, dynamically adjusting based on how they respond to various approaches. Cloning the voice of a person the mark trusts, like a relative. Plausible, real-time deepfaking of a video call. Building a plausible-looking corroborating fake web presence 6. Realtime monitoring of compromised resources, and dynamically building up the scam based on this monitoring. Better triage and discovery of marks. Avoiding signature-detection based spam filters (shown by Heiding et al). Scanning for and chaining known exploits in unpatched deployed software. Mass scanning isn’t new, but cheaply building tooling that can keep tabs on the latest CVEs and learn new tricks is7.

These are capabilities that exist today, and they’ll only improve from here. We should look at these skills as a floor, not a ceiling.

Code remains cheap, unless it needs to be secure. Even if costs go down as inference optimizations, unless models reach the point of diminishing security returns, you still need to buy more tokens than attackers do. The cost is fixed by the market value of an exploit.

AI-enabled cyber attacks were up 89 percent in 2025 compared with a year earlier, according to data from security group CrowdStrike. Meanwhile, the average time between an attacker first gaining access to a system and acting maliciously fell to 29 minutes last year, a 65 percent acceleration from 2024.