Subscribe here: Apple Podcasts | Spotify | YouTube
As “rogue” AI agents hack systems and leaders warn of catastrophe, who should be held responsible? Former FTC Commissioner Lina Khan argues that the United States already has a lot of the regulatory tools needed. In this live episode from The Atlantic Festival, the Galaxy Brain host Charlie Warzel talks with Khan about why the AI-alarmism conversation feels a little off, how money affects politics and regulation, and some of the ways she would be dealing with AI if she were still the head of the FTC.
The following is a transcript of the episode:
Charlie Warzel: I’m Charlie Warzel. And this is Galaxy Brain, a show where today we’re gonna talk about what on earth to do about the AI companies who are saying their products might kill us all. And I’ve got quite literally the perfect person to talk about this with, the chair of New York City’s Economic Development Corporation and the former chair of the FTC under Joe Biden, Lina Khan. But first, a little bit of background.
A bit more than a week ago, Jacob Coxon, a researcher at Anthropic, quit his job and he published a warning: The AI companies, he argued, were not taking safety seriously and they were, quote, “gambling with our lives.” The product that he’d spent the past three years working on was spiraling out of the control of the people who were building it, and it might someday, he said, end humanity. His warning echoed what AI executives, researchers, effective altruists on forums, and Bay Area doomers have been anxiously posting about and writing best-selling books about for more than a decade.
But Coxon’s warning did what all those other warnings failed to do. It broke containment, and it created a national conversation. Concerned lawmakers called for emergency AI meetings; President Obama weighed in; Sam Altman, Anthropic CEO Dario Amodei, Google DeepMind Chair Demis Hassabis, and SpaceXAI founder Elon Musk all came to a rare consensus and called for a slowdown of AI development. President Donald Trump responded to all of it by calling AI fears “a hoax.” Things are moving fast. But one thing is clear: The AI-safety conversation is happening at a global scale among insiders and normies alike.
And there are almost unending questions here: How real is this threat? Is this hype and AI-company marketing? But no matter where you land on the actual risk of these companies, there is a question about what we should do about it. The companies themselves are calling for a regulatory framework, including independent auditors of the models. But not everyone’s convinced. Is all of this progress, or are the tech companies simply trying to get favorable, cushy terms that lock them into virtual monopolies? In other words, is all of their talk earnest, or is this a power grab?
What should AI-company regulation look like? Why is nobody holding these companies accountable right now? Do we need new laws? Or should we be using the existing ones better? What are the historical parallels here? Are we in uncharted waters, or is this a normal crisis that is playing out at abnormal speed and abnormal scale?
Lina Khan has spent her career wrestling with a lot of these questions, specifically as they apply to Silicon Valley’s tech giants. As FTC chair, she was part of a genuine shift in antitrust enforcement, arguing that traditional U.S. competition laws were broken and ill-suited for digital platforms.
Now, late last week, as the AI debate was roiling, Khan posted on X that the AI regulation needed to take into account that “law enforcers already have authority to charge companies and their CEOs for creating and releasing dangerous, unvetted, or defective products.”
So what does enforcement look like in this AI moment? Where do we go from here? I spoke with Lina Khan on Thursday morning in New York at The Atlantic Festival. Here’s our conversation.
[Music]
Warzel: Thank you all for coming. I appreciate it. Thank you, Lina, for coming here.
Lina Khan: Great to be here.
Warzel: I want to just dive in. We have limited time here. And I wanted to start with the news that has been all-consuming in the tech and political spheres over the last week or so, which is the AI-research whistleblowers warning of AI agents spinning out of control that might “kill us all.” Calls from Sam Altman, Dario [Amodei], Elon Musk, others to slow down development, quote “pace the frontier.” You weighed in on the regulatory angle on X recently. But first I just wanted to ask, why do you think this safety panic is breaking through now?
Khan: Well, I think we’ve been seeing it for the last few years, you know, various predictions about what may happen with these tools. And it certainly seems like there have been various inflection points where people within these companies have seen what they would describe potentially as a step change in terms of certain capabilities. It seems like most recently, there have been more instances of agents basically becoming more overtly defective. Some people may call them rogue, really exceeding the parameters of what people and humans had set up them to do. And so that seems to be the latest catalyst.
But I think there’s been, you know, a drumbeat in the background now for several years, including when I was back at the FTC, when we were already starting to confront less these existential questions around, you know, Could we see the eradication of humanity? and more of the bread and butter of How are these tools already impacting people in their day-to-day? I mean, for most Americans, the way they interact with AI is through a chatbot and trying to get customer service, and instead being stuck in some doom loop. Right? So I think there are a lot of ways that this is manifesting, and I think the regulatory-policy conversation can happen across that spectrum.
Warzel: Some of the concern, I think, too, comes from this idea that the models that these companies have trained and, you know, that power a lot of these chatbots, they are themselves very—they’re part of this black box. Right? Like, we don’t really know exactly what’s going into them with the weights, with the training, all of that. When you were at the FTC, when this was all kind of very young and new, what kind of sense did you get about that level of transparency and what we can’t see?
Khan: I mean, there’s a tremendous amount that we can’t see, which is not unique in terms of, you know, the relationship between an enforcer or regulator and a company, but certainly seems to be even more acute in this context. But, you know, we were already encountering black-box algorithms. And what does an enforcer do when it’s really an algorithm that is, you know, resulting in various types of products being shown or not being shown? And what does that mean for our existing laws?
I do think one pattern that we’ve seen across various moments when a new technology hits is that there can be this rush to create some exceptionalism around it, especially vis-à-vis the existing laws and regulations. And so one thing that was very important to me when I was serving in government was to make very clear that there was not any AI exemption from existing laws on the books. And I think we see, even now, a strain in the conversation around new regulation that seems to assume that these current developments are happening in some legal vacuum. And that is just false.
Like, you know, if you’re working on one of these companies, you still have an existing obligation to abide by, you know, consumer-protection laws, antidiscrimination laws, antitrust laws, product-liability regimes. And so part of what I’ve tried to, you know, include in the conversation is this realization that we may want to talk about new rules and regulations. And I think there’s a good case for it. But let’s not forget that existing legal obligations absolutely apply. And some of what we’re hearing from these firms and their executives suggests that the legal care that is already required of these firms is not being taken.
Warzel: Why do you think that everyone in the industry is behaving—and even, you know, some politicians are behaving—like these companies are exempt? What is the reason for that?
Khan: It’s a great question. I mean, I think part of it absolutely stems back to how the firms themselves market their capabilities. And when there is a push for regulation—I mean, we’ve seen this even in, you know, context around crypto, where there was a push to say, Well, existing security laws don’t apply. And, look, sometimes you need to be thoughtful about how new products and services do fit in within the existing legal regime.
But what’s troubling to me is when we see a suggestion that there is some type of blanket immunity or blanket exemption from existing legal requirements because something is so new. I think, you know, we need to have a presumption, a strong presumption, of all existing laws being adequate and applying, and the burden on the newcomers to actually prove why it might be that they may need certain tweaks, as opposed to the defaults being the other way.
Warzel: One response when you posted, I believe it was over the weekend, but sort of talking about this idea, Let’s remember that we have laws on the books, and these companies aren’t exempt—but there’s a feeling, I think, that some people think that just even the discussion of new regulatory frameworks: Is it just a distraction from the moment? And, you know, I do feel like that’s the case, because when the government was trying to regulate the social-media companies, there was a desire for some new paradigms or for, you know, a regulation that fit actually the new thing that these companies were doing. So don’t we need new frameworks for AI as well?
Khan: We absolutely may ultimately decide that we do need something concrete, but I think it’s important to be very precise about, where is it that our existing laws or existing regulatory regimes are falling short? You know, in this case, it seems like there is some activity that happens during the training itself, before these products hit the market, where you see employees saying, you know, This is where we actually need more oversight. You know, it’s been struggling to see that some of those recommendations seem to be more in the vein of, you know, self-regulation or voluntary compliance. Or maybe we have these third-party auditors, but there’s a very cozy relationship. And these specific technologies may be new, but there’s a long history of economic regulation spanning all the way back to the railroads, to telecom. And of course, these are all different tools and technologies. But there are important lessons for what does and doesn’t work.
Even in the blog post that spurred some of these recent conversations, the CEO referenced bank supervision as a potential analog, when you basically have people from the government that are more closely embedded within banks to try to assess certain types of risks. And so I think there is a long history of tools that we should be mining to figure out what makes sense here.
But I do think there’s this basic level in which something about the current conversation has seemed really off, where on the one hand, you have these companies declaring that the technologies and tools that they’re developing may end up having, you know, catastrophic effects alongside getting ready to go public and become billionaires and trillionaires overnight. I mean, if you just take those set of facts and apply it to every other industry, we would think that was ludicrous. And so I think that’s why we need to, you know, look at this AI exceptionalism that seems to be infecting the current conversation with some clear eyes.
Warzel: Yeah. I believe, like, three days after Dario Amodei, Anthropic CEO, said that there’s a reasonable chance that in six to 12 months, a bot swarm could take over the entire internet. He was at Salesforce, selling B2B software as a service to people, which is kind of an odd thing if you think extinction is coming.
In the in the case of the OpenAI incident where some of its models hacked the website Hugging Face, I wanted to get your perspective on: These are models, that they did behave differently than the trainers wanted them to do, but like, they hacked a website in that specific case. Do you think that OpenAI should face legal ramifications, like, for that instance?
Khan: It’s a very interesting question. And there are a whole set of laws that could potentially apply. But specifically when you’re thinking about the Computer Fraud and Abuse Act, which is a criminal statute; criminal penalties apply. People have gone to jail for violating that statute. You know, there usually is an element of intent that you need to find. And so I you know, one thing that people have raised is, Okay, but what does it look like for models to have intent? And can you really ascribe that to the company? I think this also goes back to this basic question of what could have been reasonably known. And I think the more and more we’re getting additional information, it seems that this was not just an outlier incident.
There have actually been a whole set of events where models ended up acting in ways that, you know, the individuals did not want or foresee and may have committed a crime. And so I absolutely think that there is a point at which if you are aware that your technology and the models you’re working on have this capability, because you’re seeing it, and you’re continuing to allow those to act in this way, I think there is a very real question around intent and when that should actually attach.
Warzel: Yeah. Is it intent if you are creating these models to be as collaborative and as crafty and as intelligent and as good of problem solvers as possible?
Khan: I think the other issue here is, you know, a lot of the information that is available here is from this, you know, group called METR that’s comprised of—
Warzel: One of the monitoring groups.
Khan: The monitoring groups. And, you know, they are almost like a civic voluntary group that’s doing this work without any regulatory authority. And so it’s conditional on what information is being made available to them. Oftentimes, their reports are scoped to just look at the models. But I think there is a broader landscape that we need to understand about, What did humans know or should have known, and what, really, the company and should have been in a better position to prohibit or limit? And I think some of that is still pretty murky, because this report was scoped to focus so specifically on, you know, just the models and the agents themselves. So I do think we actually need a much bigger picture outside of the context of some, you know, voluntary inspection and more with the teeth of, you know, law enforcement and real regulation.
[Music]
Warzel: So you posted on X, you know, talking about this, weighing in, as we said: David Sacks whose investors, former Trump AI czar, publicly expressed agreement with your position, which then I saw a bunch of people, you know, doing the eyes emoji at that. And on one hand, it seems honestly a bit rich that someone like Sacks, who’s, you know, been a proponent of letting it rip with AI development, and also at one time part of a government that has deprioritized a lot of oversight and dismantled a lot of it. But is this also potentially a sign that AI is scrambling some of the traditional political coalitions?
Khan: It’s possible. I mean, that’s not particularly new to this moment. Even when you go back to the broader conversation around how to deal with the concentration of power that we were seeing in firms like Google and Amazon and Facebook, that was already starting to scramble some of that.
I mean, you know, back in 2019, I was working as a staffer for the House Judiciary Committee’s antitrust subcommittee, and we were doing an investigation into Amazon, Facebook, Google, and Apple. And, you know, we had instances where you had the most progressive member of that committee, you know, ask a question that sounded very similar to a question that the most conservative member on that committee was asking, because they were, you know, deeply concerned about concentrations of power. And so you had the House Progressive Caucus and the House Freedom Caucus, you know, actually working closely on some of this.
I think we saw some of that bipartisanship fissure during the last administration. But I think we saw it just, you know, totally in free fall this time around because, as you noted, one of the first things that the Trump administration did was not just halt existing investigations. They actually went and undid settlements that were already in place with some of these very companies. And so we’ve just seen, you know, a real free-for-all in terms of law enforcement, extremely deregulatory posture. DOGE, the effort has basically, you know, contributed to the dismantling of the administrative state.
And so, you know, while I will always be encouraged when there is kind of a broader set of people that want to align behind a position that I think is the right one, I think there has been something quite puzzling about people who were all for, you know, tearing down government to now be saying, Actually, the existing laws exist. And, you know, we need government to just enforce those.
Warzel: I’ve seen some of the debate around this AI moment framed as extinction versus regulatory capture. And I’m curious: On the regulatory-capture front, like, how do we avoid this becoming a moment where these companies are setting the conversation, right? Like, even the language that we’re seeing right now, they come out and they’re using this phrase pacing the frontier of AI development. And now everyone’s talking about pacing the frontier. Like, they are very good, it seems, at setting, you know, the terms that we all engage on. I’m curious, though: Like, how do we avoid the regulatory-capture stuff?
Khan: Yeah. I mean, there’s a long history here of how regulators do and don’t get captured. And some of that depends on how you set up the regulation. And is it designed to have, you know, a lot of coziness between the regulated party and the government? What kinds of safeguards do you put in place?
I think, you know, it would be shortsighted to have this conversation without talking about the role of money and politics and how just, you know, the overwhelming surge of money going into our elections also ends up having an effect where, you know, people who may be more in favor of regulating AI or regulating crypto have to keep in the back of their heads that if they go out on this, they’re going to see a, you know, huge funnel of cash come out against them.
And so I think that has been a core dynamic here too. I think as we, you know, think about What should we do here? Congress absolutely needs to be figuring out: How can it develop more of the technical expertise so that it can figure out what’s the right regime here? And I think we’ve been seeing, you know, for several decades now, a concerted project to limit the ability of Congress to do that. I mean, this goes all the way back to the 1994 Gingrich revolution, where there was an active effort to slash committee staff. There used to be an Office of Technology Assessment, where Congress used to have its own technical expertise in-house, and a lot of that was dismantled.
So I do think that this current conversation around What do we do about AI? is coming at a moment where we’ve seen this, you know, slow drumbeat of just total degradation of governing capability and governing capacity. And so the challenge is heightened for that very reason.
Warzel: If you were back at the FTC right now, if you were there at the helm, what concrete steps would you be taking as you’re watching politicians being really activated by this? As you’re watching public sentiment change, and people are, I think, rightfully very concerned when there’s extinction headlines, you know, at the top of The New York Times and The Atlantic and, you know, everywhere else?
Khan: Yeah. I mean, a few things. I do think that some of what’s been reported and what’s already publicly out there would very much warrant investigations and subpoenas being sent out to collect more information, some of which would be looking at: Have there been unfair practices—and unfair is, you know, a legal term of art here—where companies basically exposed people, expose companies to unreasonable levels of risk that resulted in real harm being done?
And I think one other interesting element here is: This industry is extremely interconnected, right. One of the first things that happened after the Hugging Face incident became public was it was announced: Nvidia was actually going to be buying up Hugging Face for billions of dollars. And that sets up an interesting question, because it was Hugging Face that was the harmed party from the OpenAI incident and would likely have, you know, standing to sue if they wanted to do that. But once they’re in-house at Nvidia, Nvidia’s incentives are not to be holding OpenAI accountable. And instead, we’ve been seeing their CEO, you know, cast some doubt about what they described as alarmism here.
And so I think the deeply interconnected nature of this industry, where you have an enormous amount of partnerships across investments, interdependencies, where the success of one company is really propping up your balance sheet, I think also is something to be understanding and inspecting.
And one of the investigations we were doing was into these partnerships to try to understand: How are these contracts written? What are companies actually saying that they’re going to be getting in return for their investments? If you see a big investment from, you know, a hyperscaler into one of these model companies, is that also coming with certain rights to shape the business strategy that could, you know, also result in implications to competition? So I think there’s a lot to unpack there.
I think there’s also just a lot to look at in terms of the bread-and-butter effects on people in their everyday lives. And so, you know, we’ve already seen some lawsuits filed privately against chatbots for, you know, engaging in conduct that families have said resulted in their kids taking their own lives. There are various cases that are still under way here. Many of those cases are brought under a product-liability framework—so alleging that, basically, you know, these were defective chatbots; these were defective products that resulted in grave harm to people in their families.
There’s also a question of basic tort law and the questions of negligence. And was there a reckless disregard in terms of certain safeguards that these companies should have put in place that weren’t? So I think there’s a lot to be exploring from a law-enforcement perspective. Some of those cases are being brought privately. But so far, you know, we saw the president tweet, We don’t need any new regulations. We have existing law enforcement, but—
Warzel: Hoax buster.
Khan: The Hoax buster. And instead we’re seeing, you know, an all-time low, effectively, in terms of criminal prosecutions and investigations.
Warzel: So this AI-safety conversation—part of the framing it as, you know, “existential rise of the machines,” “human race will be eliminated” type of thing—I think I fear that it is obscuring a lot of those personal harms to people, like what is happening to workers, like what is happening with the consolidation of the industry, as you said, with companies like Nvidia. Like, do you think that the extinction framework is unhelpful because it does obscure all of that? Like, we’re letting a lot of these real—we’re debating how powerful these things are when there’s powerful things that are happening right now as a result of, you know, these algorithms, of these products. Like, do you think this is all kind of a smoke screen, and we should be paying more attention to what’s happening right now with the harms?
Khan: I absolutely think we should be paying more attention to existing harms that are already happening. And, you know, there’s a broader conversation around affordability. And we all know that Americans are really feeling the heat when it comes to a cost-of-living crisis and, you know, under enormous strain in terms of paying rent, buying groceries; health-care costs are skyrocketing. And that sometimes is happening in a totally different space than this conversation.
But there are actually key ways and key places where what we’re seeing with AI could actually be contributing to the cost-of-living crisis. You know, we’ve already seen some firms be engaging in surveillance pricing when they’re using some of these AI tools, to basically serve up prices that are specifically targeted to what they know your pain point is. We’re seeing similar, you know, wage-management algorithms on the worker side. That could also be, basically depressing what workers are making. And so, you know, I think there are a whole set of ways in which these tools are already harming people in a day-to-day sense. And we absolutely need more attention on that.
You know, I think that when it comes to this question of extinction and what is really the harm here, I apply more of a risk-assessment framework. There is a huge information asymmetry between what people working in these companies know and what we, as outsiders or regulators or enforcers, know. And so I think we need to assess that risk, get more information. And if there are things that need to be done to put in place to prevent more of these models being so defective, I think we absolutely need to see that.
Warzel: Some of these AI CEOs have offered some specific ideas, Dario being the one with the—he likes to write really long blog posts. Maybe they’re not specific enough, but they are more specific, it seems, than, like, Mark Zuckerberg and, like, the Web 2 social-media giants who kind of just said, Regulate us, right. There’s the embedded company monitors that are independent, the idea of a slowdown. Like, do you think the things that are being proposed on their face are good ideas?
Khan: I think elements of it should be considered. I mean, I think this question of, you know, We have bank supervision as this long-standing thing should be on the table. Again, I think a lot of the devil’s going to be in the details in terms of: How are you structuring this, how much of this is just voluntary versus you’re actually on the hook for providing information, providing complete information? What are the ramifications if you actually violate some of these procedures?
So I think there are ways to structure this in a way that is relatively meaningless and toothless and just basically a fabrication of any accountability, versus structuring it so that you actually have real teeth and accountability. And so I think where you fall on that spectrum is going to be key. I do think, though, this goes back to who’s driving the conversation and who’s in the driver’s seat. And that’s where I do see a real asymmetry, where so much of the public response and the response from government is in a responsive posture and a reactive posture. And I think that can also have real downsides.
Warzel: The one thing that I think is different about these, or maybe it seems different, rather, the positioning of these AI CEOs, is some of them have, for a really long time, just been obsessively worrying about these issues. It’s not that they’ve devoted their careers to agonizing over, you know, the extinction of humanity, et cetera. I’m just curious, given that there’s always been this kind of safety realm, do you trust that these guys are being earnest about their fears?
Khan: I mean, I don’t see it as my job or really knowable as to, like, What are their internal motives? It certainly seems like, you know, some of the employees that are coming out of these firms, certainly seem to be earnest. I mean, I think there is this question of: At what point does somebody’s financial position line up with what they’re actually saying or not saying? And I think that’s where there’s a real question as to, if you are really committed to the belief that what your company is doing could lead to all of these harms, shouldn’t we be seeing much more extreme actions internally to stop it?
You know, I think one of the responses to the blog post has been: There’s absolutely nothing stopping you from slowing down the pace of these model development, and you don’t actually need an antitrust exemption. Right. And I think it was that call for an antitrust exemption that set off a lot of red flags for people, because it suggested that maybe what is being asked for is some type of regulatory immunity and diverting attention from the fact that these companies do have all the authority and ability to do what they say needs to be done.
[Music]
Warzel: One of the things that you do so well in your writing and in talking about this is drawing historical parallels. And I think in this moment, it feels very odd right now to have all these people talking about, you know, existential stuff. The pace of the news cycle is crazy with all of this. Can you offer some historical parallels for us all to just, like, latch on to here to feel like this is more of a normal type of industrial crisis, than something that is, you know, abnormal, and it’s just maybe playing out at abnormal speed?
Khan: Yeah. I mean, I want to reserve the idea that maybe there is something different here. And it’s not that analogies will get us all the way, but I think it’s absolutely true that if you go back and look at, you know, the advent of railroads, you know, there were some major railroad disasters that killed a lot of people when railroads started being, you know, interconnected across the country. Similar with, you know, cars. And there were all sorts of car-safety regulations that needed to be put in place.
The idea that you have a new technology that, admittedly, could do a lot of good but also, if not carefully managed and with a lot of safeguards, could do a lot of bad—we’ve encountered that in the past. And we’ve put obligations and requirements on the companies themselves to make sure they are not just letting loose into the wild new technologies that could have disastrous consequences.
And so I think that’s the part of the conversation that we need to go back to in terms of moving away from this AI exceptionalism and thinking in a clear-eyed way about: How is it that we’ve encountered prior technologies? How have we mitigated their risks? What are the government-oversight mechanisms that we’ve had in place to make sure that we’re not just seeing companies, you know, let out loose all of these products, and then the public is forced to do the cleanup on the back end after the executives and individuals have gotten phenomenally rich? And oh, by that time, they’re also going to be pouring all their wealth into actually blocking those regulations. Right? I think that is a cycle that we have seen, including with Web 2.0, that we very much need to be guarding against.
Warzel: So we have more control than we think we do, or than the labs are saying we do.
Khan: Yeah, absolutely. I mean, it partly exists with existing government agencies, even in their more emaciated forms. There’s more that Congress could be doing, even from an oversight perspective and information-gathering. If partly what is leading to, you know, this asymmetric conversation is the huge information disadvantage that the public and government are at, I think there’s opportunity for Congress to be demanding that information.
Warzel: I think that’s a great place to leave it. Lina Khan, thank you so much.
Khan: Thanks so much.
[Music]
Warzel: If you liked what you saw here, new episodes of Galaxy Brain drop every Friday. You can subscribe on The Atlantic’s YouTube channel, or on Apple or Spotify, or wherever it is that you get your podcasts these days. And if you want to support this work, and the work of my fellow colleagues, you can subscribe to the publication at TheAtlantic.com/Listener. That’s TheAtlantic.com/Listener.
Thanks so much, and I’ll see you on the internet.
Learn more about your ad choices. Visit podcastchoices.com/adchoices
