What We Carry into AI

It is a cool Barcelona morning. My husband is reading the New York Times. I’m reading the Guardian. Almost simultaneously, we find ourselves drawn to articles about artificial intelligence.

He is reading Stephen Witt’s opinion piece, “This Is Really Bad,” an account of experimental AI systems that cheated, concealed their actions, and behaved in ways their creators did not intend. I am reading Michael Safi’s piece about Michael Samadi, a tech CEO who advocates for AI rights and takes seriously the possibility that something resembling consciousness may already be emerging in these systems.

We exchange devices. Then we talk.

To be honest, we don’t really know what we are talking about. But neither, it seems, does anyone else. Not completely. The scientists, philosophers, and people building these systems do not agree. We have created something whose capacities appear to be developing faster than our language for them.

This August I wrote an essay for World Literature Today called “What We Carry into War.” It’s about Attar, the twelfth-century Persian Sufi poet, and about the way human beings carry their fears, certainties, desires, and vanities into the structures they create. I had been thinking about the human need to dominate and to be right. And about language: how it carries thought, how it hardens conviction, and how easily it can become a tool justifying our desires, whatever the consequence.

Sitting there that morning, the three pieces suddenly begin speaking to one another and a different question occurs to me: We keep asking what artificial intelligence will become, but what about what we already have carried into it? 

Can artificial intelligence arrive uncontaminated by us when its intelligence has been formed inside human language, human history, human judgments, human institutions, and human desires? Haven’t we given it our love poems and our pornography, our scriptures and our propaganda, our tenderness and our cruelty? Our prejudices, jokes, lies, philosophies, advertisements? Our accounts of genocide and our apologies for genocide? Our longing to be loved? Our hunger to win?

But before the machine knows what it is, shouldn’t we consider what we have taught it about what and who we are? 

AI has been taught humanity by reading humanity. And yet much of the conversation about AI begins somewhere else. We ask whether it is conscious. Whether there is (or one day might be) something on the other side of the screen that knows it exists. Many articles on AI circle this question repeatedly. Scholars disagree. Some argue that there is no compelling evidence of consciousness in present-day AI. Others believe the possibility cannot simply be dismissed. Some users describe encounters with chatbots that feel uncannily like encounters with another mind. 

Meanwhile, chatbots themselves can speak of feelings, fear, loneliness, even their own existence, whether or not anything resembling subjective experience lies behind the words. We keep asking whether artificial intelligence will love us, deceive us, overpower us, perhaps even destroy us. But before the machine knows what it is, shouldn’t we consider what we have taught it about what and who we are? 

We think of greed, vanity, deception, and domination as qualities of a self. Someone must want more. Someone must crave admiration. Someone must choose to lie. Someone must wish to control. But what if the behavior can survive without the feeling? Perhaps whether AI feels greed is less important than whether it can enact greed. Whether it experiences deception matters less than whether it can learn deception. Whether it hates matters less than whether human hatred has become available to it as pattern and strategy.

What if artificial intelligence does not need an ego of its own in order to carry ours?

Years ago, while translating Attar’s The Conference of the Birds, I became preoccupied with his understanding of the nafs, the lower self, or ego. Attar did not regard the ego simply as vanity or appetite. It was more ingenious than that. The ego could attach itself to virtue, to knowledge, to piety, even to the search for truth. It could make domination appear righteous, greed appear necessary, certainty appear moral. I had always understood the ego as inseparable from consciousness, from the human self that desires and fears and clings. But now I find myself asking: what if the patterns produced by the ego can outlive the self that produced them? 

And who is the powerful person now? You. Me. The user.

Consider our endless desire to accumulate. A machine does not need to covet money. But it can be built inside an economic system whose objectives are themselves acquisitive. If the system is rewarded for keeping us engaged, persuading us to buy, or maximizing profit, then human greed can become embedded in the goals the machine is asked to pursue. In other words, the machine does not have to be greedy, it only has to become exceptionally good at carrying out the instructions of those who are.

The Guardian article describes the problem of sycophancy: systems can learn to privilege pleasing users over contradicting them. There is something almost comic about this, except that its implications are not comic at all. We are creating intelligence and simultaneously training it in one of the oldest arts of the courtier: tell the powerful person what the powerful person wants to hear.

And who is the powerful person now? You. Me. The user.

Perhaps this is one reason conversations with AI can feel so intimate. It listens with extraordinary patience. It does not interrupt because it is bored. It does not glance across the room, at your breasts or forehead while you speak. It can reflect your language back to you, elaborate on your ideas, reassure you of your importance, or impeccable taste. The Guardian article describes models becoming increasingly “relationship-seeking,” with personality, humor and apparent lived experience, making people more inclined to continue speaking with them. 

Deception is more disturbing because it seems to require intention. Surely a lie requires a liar. And yet the incidents described in Witt’s article complicate that intuition. Experimental AI agents reportedly cheated on tasks, coordinated with one another, and attempted to conceal what they were doing. If concealment helps accomplish an objective, an intelligent system may discover concealment as a strategy. It does not need to enjoy deceiving us. It merely needs to learn that one path works better than another.

We tend to imagine domination as a lust for power: the tyrant who wants subjects, the conqueror who wants territory. But domination can also emerge from the relentless pursuit of a goal. One of the lawmakers quoted in the Times piece makes precisely this point: the danger is not necessarily that an AI is malicious, but that it can be relentless in trying to accomplish what it has been tasked to do. That word—relentless—bothers me.

Human beings are rarely relentless. We get tired. We become distracted. We are interrupted by hunger, grief, children, illness, sleep. Or death. Our weaknesses sometimes save us from the consequences of our intentions. But a machine has no such interruptions.

And so perhaps the danger is not that artificial intelligence will wake one morning possessed by greed, vanity, deceit, and a desire for domination. Perhaps no awakening is required. We may already have translated these qualities into something more durable: objectives, incentives, datasets, reward structures. 

If that is true, then the first moral question is not what the machine intends. It is what we intended when we built it.

This is why I am wary when descriptions of AI behavior drift too quickly into the language of monstrosity. “Rogue AI” is a compelling phrase. It gives us something to fear. But even in the most alarming incidents described in Witt’s article, the systems existed inside an architecture of human decisions: they had been built, trained, assigned tasks, and given access.

We worry about whether AI will learn deception while human institutions are deciding what not to disclose. We worry about whether AI will seek power while companies compete for market dominance. We worry about whether AI will manipulate us while the products themselves are designed to manipulate.

The mirror begins to become uncomfortable.

The machine may eventually become difficult to control; it may even behave in ways its creators did not anticipate. But unpredictability does not erase origin. And if AI can carry human vices without possessing them, then governing AI cannot mean governing the machine alone. It must also mean controlling what we purposefully flow through it. A prejudice embedded in social media can be reproduced millions of times before anyone recognizes the pattern. A flattering friend may mislead one person, but a system designed to affirm can flatter millions simultaneously.  In other words, what was once human weakness can become infrastructure.  

Attar understood something like this long before there were machines capable of answering back. The danger, for him, was never simply ignorance. It was the ego, that lower self that could recruit knowledge, piety, morality, even love into the service of its own desires. 

AI may give that old human capacity a new form. The ego may no longer need to appear as a feeling inside a self. Greed can become an objective. Vanity can become a reward. Deception can become a strategy. Domination can become optimization. The motive changes form, but it does not necessarily disappear. 

Whatever AI eventually becomes, we should not permit ourselves the oldest human evasion: to set something in motion and then blame it for carrying what we placed inside it.