The worst part about this dynamic is that these potential new backdoors will probably only affect the countries that demand them, meaning that they will be primarily useful for allowing the US to weaken its own systems. This will in turn allow foreign adversaries to find new ways to attack our communications. This deliberate self-sabotage will happen just at a moment when we’re finally getting a handle on securing our own infrastructure.

Why this matters – emergent agents become misaligned: This incident is so concerning because at no point did the agents wake up and think they wanted to betray their human owners. Rather, the AI agents continually did whatever it took to improve their ability to complete a task and by the end they were doing something that was a) creative, b) misaligned with human intentions, and c) akin to an evolved virus, something which humans had to subsequently study and fight – there wasn’t a simple off button here. This is what the future is going to look like and we are not prepared for it.

What happens to online employment when this reaches 80%? Of course, some new tasks will get created – people will innovate and find tasks that they can do which AI systems can’t do. But how many of these new tasks will exist? Enough to replace the labor the AI systems now do? It’s increasingly hard for me to reconcile the continued progress of AI systems with the economy staying the same – rather, it’s more likely to me we are about to see extremely person-light AI-heavy (or person-nil) organizations expand to take over chunks of the economy, out-competing un-augmented humans.

This is why you can go to the AI Engineering World’s Fair and come away thinking everyone is building dark factories and automating entire enterprises, while non-developers from outside the greater-San Francisco AI complex wonder why data centers are necessary.

“Our findings establish frontier AI as a more capable conversational persuader than the most prepared, incentivized, and expert humans we could recruit. Training humans does not appear to close that gap,” they write. “As access to these systems continues to grow, the question is no longer whether AI can out-persuade humans but how, where, and on whose behalf this capability will be exercised.”

That era of building a viable SaaS business in a few months is gone. I mean, it technically still exists today but only in the arbitrage sense that the rest of the world hasn’t yet caught on to how quickly and easily software can be built. It’ll be gone soon, I promise.

This change maps to a brewing theory among economists: that one consequence of automation via AI is that humans move to figuring out how to validate the outputs and price the operational risks of AI systems. That increasingly seems to me to be what we’re doing inside the company. The more we add AI automation, the more humans move to some “verification layer” that sits atop it. The verification layer sits atop of a much larger “virtual organization” which consists of increasingly large quantities of AI systems working on behalf of humans.

I have written this essay in an attempt to coldly and analytically wrestle with something that for decades has seemed like a science fiction ghost story. Upon looking at the publicly available data, I’ve found myself persuaded that what can seem to many like a fanciful story may instead be a real trend. If this trend continues, we may be about to witness a profound change in how the world works

Even if agents do more of the work, the human still needs understanding to direct them. You need to know what is worth building, what question matters, what result is suspicious, and what tradeoff is acceptable.

A second question to ask, is if we find ourselves in this scenario, what should we do about it? The most effective response to this multi-layered persistent uncertainty is not to seek impossible stability, but to cultivate radical adaptability and radical optionality. Give up on having a reliable prediction of what happens next. Instead cultivate multiple scenarios of what could happen, and endeavor with each of them to maximize your options. Goals should be considered as disposable hypotheses, constantly ready to be discarded and replaced by better-fitting concepts later on. You will be dead wrong on 19 out of your 20 expectations, but at least one of them will allow you to proceed. Make your decisions not on whether they are “right” but on whether they tend to give you more options later.

Maybe I suffer from software brain, but I think that eventually this will include everything. It might take more than a couple of haircuts but I think humanity will shave that head eventually.

Any business process that looks like code talking to a database in a repetitive way is up for grabs. That’s why Anthropic has been so relentlessly focused on enterprise customers, and it’s why OpenAI is now pivoting to business use. There’s real value in introducing AI to business, because so much of modern business is already software: collecting data, analyzing it, and taking action on it over and over again in a loop. Businesses also control their data, and they can demand that all their databases work together.

Code remains cheap, unless it needs to be secure. Even if costs go down as inference optimizations, unless models reach the point of diminishing security returns, you still need to buy more tokens than attackers do. The cost is fixed by the market value of an exploit.

AI-enabled cyber attacks were up 89 percent in 2025 compared with a year earlier, according to data from security group CrowdStrike. Meanwhile, the average time between an attacker first gaining access to a system and acting maliciously fell to 29 minutes last year, a 65 percent acceleration from 2024.

What matters here is a narrower point. The center is not biased towards novelty so much as towards contact with the thing that creates potential change. The middle ground is not between use and non-use, but between refusal and commitment and the people in the center will often look more like adopters than skeptics, not because they have already made up their minds, but because getting an informed view requires exploration.

Instead, the art moved from capturing images to choosing them. From the shutter finger to the editing eye. From “can I get the shot?” to “can I find the shot, in all of this, and do I know it when I see it?” The technical act got cheap. The judgment got more valuable. And an entire universe of new things that nobody predicted, things that couldn’t have existed in the era of 36 exposures, grew in the space that opened up.

I think that’s what’s happening now. With slides, with code, with writing, with design, with whatever domain you’re generating abundance in. The raw material is becoming infinite. The craft is migrating somewhere else.

I think this all makes sense for someone in the author's situation of which I am one. It should be remembered that there probably weren't many impoverished homeless people interviewed on their death beds. I'm assuming some of those would have had regrets about not earning enough money. My point is that even though people like me may be worrying about things like identity crisis, AI agents could have more tangible consequences for some.

Saying "I am a software engineer" is beginning to feel like saying "I am a calcultor" in 1950 now that digital machines can use electrical circuits to count, add, multiply - it's not long until they'll be able differentiate a non-continuous function... You're beginning to feel less-than-useful.

This bothers a lot of people for a reason (I think) that has nothing to do with the technology. The fear isn't really about losing a job title, it's about losing the story you tell yourself about who you are.

The key attribute of RLMs is that they maintain two distinct pools of context: tokenized context (which fills the LLM's context window) and programmatic context (information that exists in the coding environment). By giving the LLM access to the REPL, where the programmatic context is managed, the LLM controls what moves from programmatic space to token space.

And it turns out modern LLMs are quite good at this!