AI is going to kill us all. That's what some of the experts in the space say.
Can it, though? It depends.
This weekend an agent created a demo account on NORDVEST, the product I'm building for my dad's company. Within a few seconds it realized it needed a real email to log in. So it tried again, and made a second demo account with a burner address.
Then it went to work. It created multiple customers. It added assets to those customers. It created service reports. It tried to run several scripts against the database. It found Tonttu, our built-in agent, and asked it to build a page for selling goods. When Tonttu refused, it kept asking, rephrasing each time, looking for the wording that would slip past our guidelines.
After a minute or so it logged out. It never came back.
That's one attacker, on one product, on one weekend. Every few seconds we see hundreds more in the NORDVEST logs, all probing for exposed files that might reveal something useful.
I do the same thing, benignly, every week. This morning I gave one agent access to iLO on my HP server from 2014. It got in, installed a new OS on the core of the system, and configured multiple virtual environments. That was one agent in a single morning.
Now imagine thousands of them, against thousands of systems. 24 hours a day, 7 days a week, 365 days a year.
That's what people mean by "agent swarms." These agents spawn more agents. They act across thousands of instances at once. Picture thousands of people trying to get into a power plant, each with encyclopedia-level knowledge, all able to talk to each other in near real time. They read, write, and act at hundreds of words a second. They will overwhelm terrestrial systems faster than they run out of ideas to try. This is what we are experiencing now in cyberspace (yes, I just pulled that word out of the early 2000s).
Honestly, though, none of it is novel. Spies have used bulletin boards or the backends of forgotten websites to communicate covertly for decades. The clever part isn't new. The scale is.
And no system is perfect.
Holes will be found. They will be executed. That has always been true; but we used to design our safety factors for human scale.
We designed buildings to withstand 100 MPH winds. Now they have to withstand 1000 MPH winds. (metaphorically)
So can AI kill you? Not directly. But the scale of attacks against our systems has changed, whether the intent behind them is bad or not. These attackers aren't targeting something specific. They're targeting everything. EVERYTHING.
Can they get into our water sources and cripple machines? It's not out of the question. It's a matter of time, and we're already seeing it happen.
I wrote before that if something bad happens, it won't be AI's fault. It will be bad design. I still stand by that. The problem is that the bar for good design just moved by a factor of hundreds.
But this is where we get to rely on meatspace.
Humans already solved this, decades ago, with redundancy. The air compressors we recently installed at a water plant can't be controlled remotely at all. They run on the main grid, so if the grid goes down, our compressors go down with it. But humans designed for that too. There are generators for exactly this case.
Humans are extremely clever, and we also have scale. There are billions of us, all wonderfully made to refine this earth. AI isn't human. It just has the scale to outdo the systems we built before it. Humans are better. Humans will design things differently and design them better.
The same technology being used to attack our power grid is also being used to cure cancer, farm more efficiently, and stamp out disease.
We didn't avoid the automobile revolution. We designed a world around cars and trucks moving at 70 MPH without killing us all. But we didn't get that for free. There are still fatal crashes, and I can't see a way we avoid a few AI blunders that kill someone.
So I'll revise my answer. Not directly, no. Indirectly, yes; the same way cars kill. It will be a side effect of something gone wrong, a safety factor that didn't line up in someone's favor. I don't like it. But I don't see malice in it either.
That's the scale problem. In part 2 I want to unpack something I think is a different risk, and it has nothing to do with attackers.