For many years, researchers and science fiction authors have forecast a future in which AI systems drastically increase the pace of overall scientific and technical progress, making things move so quickly that it is hard for humans to keep up.

They call this the singularity. I believe that this year we have gained convincing evidence that it has now begun.

Why this matters – emergent agents become misaligned: This incident is so concerning because at no point did the agents wake up and think they wanted to betray their human owners. Rather, the AI agents continually did whatever it took to improve their ability to complete a task and by the end they were doing something that was a) creative, b) misaligned with human intentions, and c) akin to an evolved virus, something which humans had to subsequently study and fight – there wasn’t a simple off button here. This is what the future is going to look like and we are not prepared for it.