Anthropic v. The Pentagon: A Battle of Values

The fight over AI and how it is used is, like AI itself, a reflection of who we are.

I’m am co-publishing my notes for CAFE Insider right here on my Substack, so you can get them immediately!

---

Trump claimed to have “fired” the AI startup Anthropic last week after negotiationswith the Pentagon imploded because of the company’s internal red lines on how its technology could be used. Specifically, Anthropic wanted to include as part of any contract with the Defense Department (sorry, I refuse to call it the War Department) a caveat that Anthropic’s artificial intelligence would not be used to surveil Americans or to create fully autonomous weapons. Defense Secretary Pete Hegseth balked at these restrictions, ending the negotiations and immediately labeling Anthropic a “supply chain risk,” prohibiting Anthropic’s technology from being used in any defense contract work, even by other companies. Meanwhile, OpenAI’s chief executive Sam Altman – a nemesis of Anthropic’s chief executive, Dario Amodei – immediately jumped into the void and announced that it had sealed a deal with the Pentagon to allow its technology “for any lawful purpose.” But OpenAI’s own backtracking since then, combined with the boost in Anthropic’s users and market share, suggests that when it comes to AI, a values-driven approach might also be the most profitable one in the long run.

To understand the current AI showdown, we should rewind to the original falling out between Altman and Amodei. (I was not aware of this tea until I began researching this piece, and I am here for it.) Amodei was the vice president of research at OpenAI until late 2020. In June 2020, OpenAI had released GPT-3, one of the first large language models (LLM). Amodei noted the technology’s “emergent qualities” and advocated slowing down the pace of development so that safety considerations could be taken more fully into account. This created a rift with Altman, who wanted to commercialize the technology quickly. The dispute ultimately led to Amodei, along with several other OpenAI employees, leaving to form their own startup, Anthropic – so named because of its focus on humans, social impact, and accountability. Since then, there has been no love lost between the two company heads, who have taken shots at each other over their different visions for the future of AI: At a recent summit in India, Altman and Amodei were the only two attendees who refused to raise their hands together in a show of solidarity. (They were willing to touch elbows.)

Amodei’s prioritization of the safety implications of AI is laudable in light of the huge amounts of money at stake and the first-mover advantage when it comes to tech startups. As we have seen with other technological innovations, the nature of digital technology and the rush to capture the market can lead to both companies and the public to underestimate, and even to be blind to, the negative externalities of the product. Take, for example, social media. When Facebook overtook MySpace in worldwide traffic back in 2008, it seemed like it could only be a good thing. I mean, connecting billions of people around the world, and for free? The apparent ability to help topple dictators and fuel democratic protest movements, like in the Arab Spring and the color revolutions? What’s not to like? Almost two decades later, in a fractured information ecosystem filled with disinformation that is weaponized by demagogues worldwide, the answer is, a lot. Unfortunately, once technology is unleashed in a form that saturates public life, it’s hard to put the genie back in the bottle.

This lesson is even more important when it comes to AI. One of the areas I have been delving into in my research into complicity is the concept of moral disengagement created by the social learning psychologist Albert Bandura. Moral disengagement is basically a cognitive process that helps us rationalize or justify decisions that result in harm to others. There are eight mechanisms, but three of them are especially salient when it comes to emerging technology. The first is dehumanization, which strips people of their human qualities, making it easier to remove empathy from decisionmaking. We know how this works in the context of things like genocide and human rights abuses, but it also happens when people are reduced to abstractions, like “users.” The second is diffusion of responsibility, where decisions are so dispersed or bureaucratized that no one is responsible for what happens – I wrote about how that can work in the immigration context, but it is also applicable to decisions made by algorithms, not people. And then there’s displacement of responsibility, which basically puts the locus of responsibility somewhere else, for example, “I was just following orders.” Facebook fell back on this one when it came to disinformation, claiming that it was users’ behavior, not its product, that was driving the toxic content on the platform.

AI has the potential to embed this kind of moral disengagement into the technology itself. That has the potential to create catastrophic consequences when it comes to fully autonomous lethal weapons – weapons that, once activated, can identify, select, and engage targets without further human intervention. To see why, consider certain “smart” weapons we use already, like unmanned drones. Right now, even the decision to conduct a remote drone strike on a target involves hundreds of decisions weighing things like collateral damage, the value of the target, the balance of risks and benefits, ancillary military and diplomatic considerations, and more. Imagine outsourcing all of these decisions to a large language model – particularly one that might be instructed to ignore, to use Hegseth’s words,”stupid rules of engagement” (i.e., the laws of war). Reducing humans to data points is the essence of dehumanization, and placing the power to destroy them into the hands (brain?) of AI is simultaneously a diffusion and displacement of responsibility. Who, exactly, would be accountable for the consequences once a fully autonomous weapon is “activated”?

These kinds of questions should be front of mind for all of us, especially in light of the war in the Middle East that the Trump administration has instigated. Fortunately, it seems like it is: Since OpenAI announced its deal with the Pentagon, Anthropic’s Claude has jumped to number one in Apple’s App Store as the most downloaded chatbot app across 16 countries. Outside Anthropic’s building, supporters chalked messages of support on the sidewalk. One employee even left OpenAI to join Anthropic. In response to the public backlash, Altman has expressed regret for announcing the agreement too quickly, stating that OpenAI has since added guardrailsto prevent its technology from being used for mass surveillance. (There was nothing publicly stated with regard to fully autonomous weapons, but the Pentagon has apparently agreed to have OpenAI employees work alongside government personnel on classified projects.)

Melvin Kranzberg’s First Law of Technology states that “Technology is neither good nor bad, nor is it neutral.” This is a way of saying that technology does not inherently have a moral compass – something that is especially true of AI, which reflects the collective conscience of the humans that create it. The battle over AI, and how it is used, is one about values. We can vote with our pocketbooks, if nothing else, to make sure those values are ones we can – literally – live with.