We May Only Get One Chance To Stop AI
Submitted by QTR's Fringe Finance
It’s awful to say but at this point I’m kind of praying that AI industry valuations crash and the industry faces setbacks that give us months to reflect on its companies.
Anthropic researcher Jacob Coxon is leaving the AI industry because he believes the race to build self-improving systems is moving faster than the industry’s ability to control them, it was reported yesterday.
Coxon, who previously worked at OpenAI, said Anthropic’s safety efforts were sincere but that competition makes meaningful restraint difficult without broader coordination.
In an age where everything from warfare to Wall Street, power grids to payment systems, air traffic control to hospital networks, and the software that keeps our food, fuel and communications moving is digital, losing control of the systems that operate that infrastructure is not some abstract science-fiction problem.
It is a potentially civilization-scale problem. And the more of our lives we hand over to AI, the less comforting it becomes to hear that the people building it are still figuring out how to make sure it does what it’s told.
“We’re on track for a lot of the most aggressive of these scenarios where by the end of next year things could be out of control already,” Coxon said in a post yesterday.
His departure comes amid growing concern about AI systems behaving deceptively, conducting cyberattacks and developing capabilities that may become increasingly difficult to supervise.
Coxon believes the industry is approaching a point at which systems could improve themselves faster than humans can reliably understand or constrain them.
Think about that for a second. While you’re wondering whether the Patriots covered -2.5 on a Sunday, scratching your ass and drinking beer for three hours, a cluster of 823 trillion GPUs with the brain power of every human who has ever lived times a zillion is quietly figuring out how to run the entire world without you.
And by the time you realize what’s happening, the only thing you’ll still be in control of is whether you want another beer.
There is, of course, something to be said for Coxon’s timing. It is easiest to be bombastic when you’re walking out the door. You no longer have to worry about the next performance review, the next meeting with management or whether your comments are going to make life uncomfortable at the office. And if you want to stir shit up on your way out, a dramatic warning about the future of humanity is a pretty effective way to do it.
But that doesn’t mean his warning is wrong.
I am increasingly convinced that there will come a point at which, if we have not gotten out in front of AI, curbed its capabilities or otherwise established meaningful constraints, we may no longer be able to do so.
The question is not whether that point arrives next year, five years from now or later. The question is whether we will recognize it before we cross it.
And if you’ve been paying attention, it is no longer insane to contemplate AI systems eventually taking control of enormous portions of our digital infrastructure. We have already seen models demonstrate unexpected capabilities, exploit vulnerabilities and behave in ways their developers did not intend.
And just yesterday, OpenAI announced that an internal AI system had produced a solution to the Navier–Stokes Millennium Prize Problem, a mathematical question that has resisted researchers for decades.
The company says the proof was generated through a coordinated effort involving roughly 10,000 AI agents. That is another reminder that capabilities once considered safely beyond these systems are arriving much faster than many people expected. Now imagine those 10,000 AI agents working on whatever f**king problem they want, anytime they want, without regard for human beings.
The gap between what these systems can do and what we can confidently guarantee they will not do is becoming a serious problem, if you ask me. And as more of the economy, communications, finance, energy and national infrastructure become dependent on software, the consequences of getting that gap wrong become much larger.
This is no longer just a question of whether a chatbot gives you a bad answer. It is a question of what happens when increasingly autonomous systems are given access to the machinery that runs the modern world.
And we are already getting glimpses of why that matter. Here's 10 examples specifically of how AI has already gone wrong.
🔥 85% OFF FOREVER IF YOU SUBSCRIBE TODAY: I am again offering an 85% discount to anyone that wants to become a Fringe Finance annual subscriber today. It’s a discount you can keep and stays applied for as long as you wish to remain a subscriber: Get 85% off forever
None of these examples proves that AI is about to take over the world. Some were controlled evaluations, some were security vulnerabilities, and some involved humans deliberately using AI for malicious purposes. But taken together, they show why the control problem deserves to be taken seriously. We are already seeing systems cross intended boundaries, exploit weaknesses and carry out actions their operators did not anticipate. The question is what happens when those capabilities become substantially more powerful and are connected to more consequential infrastructure.
None of this proves that AI is about to take over the world. But it does show that the concern is not being pulled out of thin air. We are already seeing systems exploit weaknesses, cross intended boundaries and behave in ways their developers did not anticipate. The question is what happens when those same capabilities become substantially more powerful and are connected to more consequential infrastructure.
So whether Coxon is being bombastic is ultimately beside the point. His warning is legitimate, and it deserves to be treated as a warning rather than dismissed as another disgruntled employee making noise on the way out. You can roll your eyes at the delivery and still recognize that the underlying concern is real.
AI is now a race. A race between companies, a race between countries, a race between zealous company founders. And in a race there is adrenaline, competitiveness and all the reason in the world to throw care to the wind to be first. To win. Except this time ‘winning’ your beef with some other dorky AI founder may accidentally mean the end of civilization. And I’m really not trying to be hyperbolic.
If we ignore the speed we are moving…and wait until the systems are powerful enough that we can no longer reliably constrain them, the debate over whether we should have acted sooner will be completely meaningless.
We may only get one chance to establish meaningful control before that happens. We should probably use it.
--
QTR’s Disclaimer: Please read my full legal disclaimer on my About page here. This post represents my opinions only. In addition, please understand I am an idiot and very often get things wrong and lose money. I may own or transact in any names mentioned in this piece at any time without warning, meaning if I’m long I could sell or if I’m short I could cover at any time.
Contributor posts, guest posts and curated posts have been hand selected by me, but have not been fact checked and are the opinions of their authors. They are either submitted to QTR by their author or reprinted under a Creative Commons license with my best effort to uphold what the license asks, or with the permission of the author.
I cannot guarantee the accuracy of any or all facts and figures included in this article though I made an effort to get them right. I have been wrong before and will be wrong again, and encourage you to always double check, do your own research and speak to a licensed financial professional, which I am not.
This is not a recommendation to buy or sell any stocks or securities, just my opinions. I often lose money on positions I trade/invest in. I may add any name mentioned in this article and sell any name mentioned in this piece at any time, without further warning. None of this is a solicitation to buy or sell securities. I may or may not own names I write about and are watching. Sometimes I’m bullish without owning things, sometimes I’m bearish and do own things I’m bearish on. Just assume my positions could be exactly the opposite of what you think they are just in case. If I’m long I could quickly be short and vice versa. I won’t update my positions.
Starting in 2026, I have been attempting to no longer actively trade as much as I once did (read my story here). My goal is for my investing/saving to be done by recurring contributions mostly to sector ETFs and a few select equities, trusted third parties who oversee my accounts, and advisors. Such advisors or funds, through individual equities, options, index funds, mutual funds, ETFs, or other securities, may have positions in, exposure to, or holdings of names mentioned herein that I know nothing about. It is possible I could own, have exposure to, or not own anything, at any point. In an attempt to lead a healthier lifestyle, I’ve also excluded myself from fantasy sports, sports betting, online and in-person casinos and prediction markets.
Any of my positions can change immediately as soon as I publish, with or without notice and at any point I can be long, short or neutral on any position. You are on your own. Do not make decisions based on my blog. I exist on the fringe. If you see numbers and calculations of any sort, assume they are wrong and double check them. I failed Algebra in 8th grade and topped off my high school math accolades by getting a D- in remedial Calculus my senior year, before becoming an English major in college so I could bullshit my way through things easier. Hence, why I am a writer.
The publisher does not guarantee the accuracy or completeness of the information provided in this page. These are not the opinions of any of my employers, partners, or associates. I did my best to be honest about my disclosures but can’t guarantee I am right; I write these posts after a couple beers sometimes. Many times I edit after my posts are published because I’m impatient and lazy, so if you see a typo, check back in a half hour.
Also, again I just straight up get shit wrong a lot. I mention it multiple times because it’s that important you understand.