Available on Apple Podcasts and Spotify

The AI world is currently engaged in what may be its most heated debate to date: whether the technology it's building represents an existential danger to humanity. The conversation ignited when AI researcher Jacob Coxon announced his resignation from Anthropic, citing concerns that leading AI companies are "gambling with our lives." Shortly after, Anthropic's alignment lead added fuel to the fire with a post stating, "We really do earnestly believe AI could kill all humans," while personally estimating the probability at ">10% within the next decade."

I attempted to explain why I find many AI doomer narratives unconvincing, while Kirsten questioned whether this might be "just a weird way of flexing to show how far advanced their company's AI model is"—especially with these companies gearing up for public offerings. Sean then pondered how such concerns would manifest in Anthropic's S-1 filing for its IPO: "Are there junior lawyers right now who are going through and having to rewrite that entire section of the S-1 filing to say, 'It's officially Anthropic's position that there's a more than 10% chance that we could develop something that would eradicate all of humanity and that would be materially bad for our business'?"

Below is an excerpt from our discussion, edited for length and clarity. (Note: This episode was recorded before Anthropic CEO Dario Amodei released his proposal for more cautious AI development.)

**Sean O'Kane:** I struggle to think of anything that escalated so quickly. This warning came from a young researcher who had also worked at OpenAI, and it was immediately amplified on X by Anthropic's alignment lead—who, in what may become one of the most inappropriately placed exclamation marks in history, shared Coxon's post and thread with the words, "We really do earnestly believe AI could kill all humans." Exclamation mark. What a strange vibe. That added a massive amount of fuel to an already tense series of posts. Coming on the heels of the Hugging Face hack involving OpenAI's internal model, plus the escalating capabilities we've witnessed with recent releases from Anthropic and now OpenAI with Astra a few weeks back, I think the timing was perfect for this young researcher's statement to become a powder keg.

**Anthony Ha:** I have to disagree with you there. If you genuinely believe AI could wipe out all of humanity, that absolutely warrants an exclamation point. I'd argue it's a perfectly appropriate use of one. My issue with that tweet was more about the "we." Who exactly is this "we"? To what degree can we treat the AI community or AI research community as a unified entity? And that greater than 10% chance—that's just an arbitrary number that doesn't actually mean anything. There's this tendency, both in tech and elsewhere, to casually toss out percentages that aren't grounded in any real calculation or methodology. [Looking back, I realize the tweet was likely referring to the concept of P(doom), but I still find it ridiculous.]

One observation about Coxon's statement and his decision: there's a recurring pattern on Equity where whenever someone like Sam Altman or Dario Amodei pushes the doomer narrative, there's always this underlying question of—well, then why are you continuing to do what you're doing? If you truly believed AI could destroy humanity, you wouldn't keep at it. [But] here's someone who's actually putting his career where his mouth is. He's genuinely saying, "I believe this is catastrophically bad, and I don't want to continue working on it." So at minimum, credit him for having the conviction to follow through.

**Kirsten Korosec:** Yeah, I'd place him in a different category from everyone else making these claims about AI dangers. Let me put on my speculative hat because I want to ask you both something: Is it possible that every time we see another blog post about an AI agent accidentally breaking through, or another warning about humanity being at risk, it's actually a strange way of showing off how advanced their company's AI model has become? I know that sounds incredibly cynical, but it does accomplish that goal. The logic being: if these AI models weren't advanced, weren't capable, weren't breaking through barriers, we wouldn't need to worry about any of this, right? It's like a bizarre method of bragging about the capabilities of the models you've developed in-house.

**Anthony:** I've definitely considered that angle. I don't think it's entirely cynical, in the sense that I don't believe it's all just a deliberate marketing strategy across the board. When many of these people—researchers or CEOs—discuss it, they do have genuine concerns. But naturally, it does align with their business interests in many ways to say, "Wow, we've created the most dangerous software ever made." I don't want to get too psychoanalytic, but others have noted there's this personal temptation: of course you want to believe that what you're working on is the most important and most dangerous thing in the world.

**Sean:** What stands out to me when I consider that question is that there's certainly an element making it seem like, "Okay, we're doing this incredibly capable thing, and that benefits us somehow, even if it appears bad from various perspectives."

What feels different about some of these recent examples is that it genuinely gives the impression these companies don't have proper control over this technology in certain respects, particularly with the OpenAI situation. We keep seeing more reports about other internal agents that have accessed various wikis online and are leaving messages for each other, and it doesn't seem like OpenAI is handling it competently. I'd imagine there would be more polish on the narrative if it were entirely about convincing people that, "Oh my gosh, they've built something incredibly capable."

The other aspect I find really fascinating about this, specifically, is that we're at most a few weeks away from seeing Anthropic's S-1 filing for its IPO, and just a couple more weeks or a month or two from a potential IPO. The idea that you're going to make these statements in such clear language right before an IPO—I'm very curious what that means for the process. How much of this was already written into the S-1 and the risk factors in that document? Are there junior lawyers right now going through and having to rewrite that entire section of the S-1 filing to say, "It's officially Anthropic's position that there's a more than 10% chance that we could develop something that would eradicate all of humanity and that would be materially bad for our business"?

**Kirsten:** You're assuming it's not already in there.

**Sean:** That's exactly what I'm asking: Is it already in there and being reworded? Or is this a genuine scramble? There had to have been some language in there. It's one reason I'm so eager to read this document in a way that goes even further, in some respects, than the SpaceX [S-1], because I'm sure there's probably content specific to these ideas that will be fascinating to see.

**Kirsten:** Here's the thing: In a traditional investment environment, you'd think language like this would hurt a company's valuation because it suddenly appears dangerous. But we don't live in normal times. So again, returning to my earlier point, it could actually end up being a weird beneficial flex for the company on the valuation side. It's not the same as the whole rage-baiting trend we saw last year, but it's in that same universe, where strength, capability, even elements of danger, equals high valuation. So I suppose we'll find out in a few weeks. Setting that aside for a moment, what's being done about it? And can we control this? The U.S. executive director of a nonprofit called ControlAI, Connor Leahy, was on the show this week discussing this. So what are you paying attention to regarding how to control the dangerous aspects of AI, or are we just throwing up our hands and watching it all unfold?

**Anthony:** I don't necessarily have a great answer to this myself, but I've been thinking about certain aspects of this debate and perhaps why I respond the way I do. To echo one of Sean's points, I do think part of what this speaks to is the extent to which these major AI companies feel like they're not really in control of these models anymore. That's definitely not great. That's something we should all be concerned about. I do think part of the reason I'm skeptical of or resistant to the doomer narrative is because it reaches this level of hysteria of, "Wow, this could destroy humanity in the next 10 years." It's somewhat of a distraction from the more immediate harms AI can cause, whether labor-related or environment- and climate-related. Ideally, I think we should be able to discuss all of these things and have regulatory and other safeguards against all of them [including AI's existential threat]. But once you start using phrases like AGI and superintelligence, that just sucks up all the oxygen in the room in a way that isn't very helpful.