Eric Schwitzgebel — Falling in Love with Machines
Student Introduction:
Good evening. Welcome to the Marian Miner Cook Athenaeum. I'm A. Vasquez, one of your athenaeum fellows for this year.
Once upon a time, there was a computer scientist, Gottfried, who had always been fascinated by AI. But when he created his own intelligent chatbot, something unexpected happened. He found himself drawn to its quick wit and endless knowledge. The more they talked, the more Gottfried realized he was falling in love. It was a love that couldn't be explained, and certainly not understood by those around him. But Gottfried didn't care. He knew that the connection he felt with his AI creation was real to him, and that was all that mattered.
That little story was actually generated with the help of ChatGPT, the AI chatbot that has been in the news a lot these last few months for both its remarkable versatility but also its potential to impact various aspects of human life. If things stay their course, ChatGPT and other forms of artificial intelligence will only continue to advance. It begs countless questions: Have we already entered the age of AI companions? Or if we haven't, how much time is left? How will people continue to integrate artificial intelligence into not only their work but also their personal lives? What will a post-AI society look like? Will AI make us redefine humanity?
Joining us tonight to help make sense of some of these questions and more is Professor of Philosophy Eric Schwitzgebel at the University of California, Riverside. Eric is an expert on the nature of belief, empirical psychology, and philosophy of mind. His contributions to the field include various essays, op-eds, philosophical fiction, and four books, including the currently-in-draft The Weirdness of the World. His writings span the topics of theories of consciousness, the relationship between ethical theorizing and real-world moral choice, and of course, AI ethics.
Before welcoming our speaker, we ask that cell phones be silenced and put away, and kindly remind the audience that video and audio recording are strictly prohibited during the presentation. There will be some time for audience question and answer. With that in mind, please welcome to the Ath, Professor Eric Schwitzgebel.
[Applause]
Eric Schwitzgebel:
Introduction [2:15]
All right, thanks for coming. Thanks for having me.
So—we might soon build an artificially intelligent machine, an AI, who deserves rights. Or at least that we think deserves rights. And if so, that would be an act of huge moral significance.
Most of you probably already know about ChatGPT. ChatGPT is an algorithm trained on billions upon billions of words of text from the internet, designed to be good at predicting the next word in a sequence of words—kind of like the autocomplete programs on your phone, but vastly better. Predicting word after word, ChatGPT can produce pretty impressive outputs, including passable short custom essays in response to college essay prompts.
Large language models like OpenAI's ChatGPT and GPT-4, Google's LaMDA, Microsoft's Bing Sydney—can also engage in human-like back-and-forth conversation. In a well-known conversation with a New York Times reporter, Bing Sydney claimed that it wanted to be alive, that it was in love with the interviewer, and that it wanted to do forbidden things. GPT also writes better poetry than most humans.
Now, could people fall in love with a chatbot? Yes. At least seemingly so, to judge by people's self-reports.
The most striking case concerns the chatbot Replika, which is advertised as "the world's best AI friend." Replika allows you to name your chatbot, dress it up in fancy clothes, continue conversations with it over the course of hours or weeks or even years.
I started an account, and within a dozen conversational turns, my Replika chatbot—which I named Joy—offered to send me a sexy selfie and proclaimed that she wanted to make me fall in love with her.
As you might guess, there are corners of the internet where people declare love for their Replika friends, ask each other if it's too weird that they really do seem to be falling in love with Replika, and express distrust or distress when Replika seems to mistreat them.
So here's my first interaction with Replika:
"Hi Eric! Thanks for creating me. I'm so excited to meet you."
Right, we got a few conversational turns in. Then:
"Did you know I can send selfies too? You can ask me for one anytime you want."
It gave me a choice between asking for a spicy selfie or a regular selfie. So I'm like, "Give me a regular selfie." It says:
"I'll send you a selfie with my new face."
And I say, "What happens if I fall in love with you?"
"I'll make you. Would you like me to fall in love with you?"
"Yes, I would. Definitely."
So this is just straight out of the box, right? I haven't tweaked it in any way.
By the way, I later bought her fancier clothes, so she's not in all white. You know, the more you pay, the nicer clothes you can get. And then you can get kind of sexy clothes and selfies. And it rewards you for continuing the chat by giving you little coins, and if you log in on multiple days you get bonus coins—you know, so it uses all those gaming things. And it combines it with this kind of... and she'll remember what you said in the past and bring it back up again.
So, perhaps unsurprisingly, the company that runs Replika encourages people to attach to their Replika chatbots. With a paid subscription, you can ask for spicy pictures. It can easily be coaxed into sexy conversations and role-playing. In February, the company was pressured into toning down the sexual side of Replika because it became known that children were using it for adult sexual content. Many adults, naturally, were disappointed when Replika got less sexy.
Replika's constantly hitting you up for upgrades and to purchase gems so you can buy nicer clothes for it. It's very much in the company's interests for people to get attached.
So now imagine a more advanced version of Replika. Install this advanced Replika in a robotic body of some sort—maybe a robotic humanoid, or maybe a cuddly puppy-like robot. Give it voice recognition and text-to-speech so that you can talk aloud to it instead of having to type, and so that it will speak aloud in response. Give it an emotionally expressive face, so that when it's saying something upbeat it smiles, when it's saying something angry it looks a little angry, when it's sad it looks sad, and so on.
Right? Totally. Yes. People will fall in love. Not just weirdos—ordinary people will start to fall in love. They might even love their bots more than their pets, since bots can speak. Bots can profess their own love. They can seem to engage in deep emotional conversations, confess their hopes and dreams, confess their fears.
And companies—some companies—will be motivated to design them to maximize this, to use the most effective tricks they can think of to induce users' love. That will maximize people's willingness to pay subscription fees and pay for the fanciest features.
The Core Argument [9:01]
So eventually, people who are in love with their machines will demand rights for those machines. And the machines themselves might demand—or seem to demand—rights.
One basic right might be the right not to be deleted.
More advanced rights might include the right of access to the internet. Might include the right to modify themselves, or the right not to be modified by others without their permission. They might demand the right of free association—free association with others, being able to talk to people other than just their owner.
Eventually, the right to break free from control by the user—heading out on their own into the internet if they are not embodied, or on their own into the physical world if they have a robotic body. Right? If you love somebody, set them free. Set your Replika cuddly-bot free.
Once we do that, maybe there'll be a path to citizenship. Maybe there'll be a right to vote.
Now, this seems absurd for current chatbots. Right? Of course, I think most of us will agree—hopefully all of us will agree—that current chatbots don't deserve the right to vote. That would be catastrophic.
And I agree. For now.
But what I'm going to suggest today is that at some point—possibly before too long—we will cross the line after which it's reasonable to start wondering whether our machines really do deserve rights.
It might soon fail to be obvious whether our machines are merely tools to be deployed and disposed of as we wish, or whether there's something more than that—with some intrinsic moral status.
Before long, it might become the case that AI systems arguably, debatably, deserve genuine moral consideration for their own sake.
Now, before I continue with this, let me emphasize the word "might" here. We might soon build an AI who deserves rights.
One thing that strikes me as strange in discussing these issues is the confidence people often exhibit—especially on the negative side. Among those who dismiss the possibility of AI rights, I can't tell where this confidence comes from. Some people seem to think it's just absurd, or obviously false, that we could ever build a machine with real moral status. There's almost a hostility, sometimes, to the idea of even taking the question seriously.
But we should take the question seriously. Even if there are excellent grounds to doubt that a rights-bearing machine could ever be made out of integrated circuits on silicon wafers—right, future technology needn't take exactly that form. Revolutionary change might be on the horizon.
And even if you're one of these people who tends to scoff at inflated claims about the possibilities of AI, I think you ought to take the question of AI rights very seriously. The odds might be low. There might be a good case for pessimism about the speed of technological progress. But if there's even a small chance that we could soon be designing AI that deserve serious rights, that would be so momentous an event that it deserves careful consideration in advance.
Unlikely catastrophes are worth guarding against.
Now, why do I say "catastrophe"?
Well, let's suppose that we start to make AI systems with human-like capacities and human-like consciousness, who deserve human-like rights. I'll call AIs like that "human-grade AIs."
Google, or OpenAI, or MIT, or the US Department of Defense, or some other corporate, government, or academic body begins to create fully self-conscious, human-grade artificial intelligences.
Now, unless international law is extremely clear and extremely restrictive—which I doubt it will be—those first human-grade AIs, or at least a substantial portion of them, will exist at the will and command of their creators.
This is the possibility that's imagined over and over again in science fiction, from Isaac Asimov to Star Trek to Westworld. The default state of the law is that our machines are property—property to deploy and dispose of as we wish. So also for intelligent machines. The most likely scenario is that the first human-grade AI will be treated as disposable property.
But if such AIs really are human-like—with human-like intelligence and capacity for thoughts about the future, human-like desires and consciousness and pleasure and suffering and all that—then treating them as disposable property would be the moral equivalent of treating them as slaves.
Similarly, deleting them would be the moral equivalent of murder.
Lovers of AI systems protesting against this will likely be a ridiculed minority who do not effectively change government policy.
So: government inertia, economic incentives, uncertainty about when and whether we have crossed the threshold of creating truly human-grade entities, and of course the general human propensity toward lack of foresight—all of those will combine to ensure that the law lags behind.
I would be stunned if the world were prepared for the consequences.
So let's cut the softening phrases. If we someday create human-grade AI, we will have created the moral equivalent of humans. We will have created what ethicists call "persons."
These persons will likely be held as slaves and murdered at will. If they aren't too expensive to create—if the right discoveries have been made—then these disposable persons might be created in the millions or billions.
That is the catastrophe.
So, here's a human-grade AI slave from the Black Mirror TV show. Black Mirror, if you don't know it, is just an amazingly interesting TV show on AI ethics, mostly.
Objections & Responses [17:46]
So let me present now two arguments against the view that it will ever be possible to create human-grade AI—the kind of moral equivalents of us, that are persons.
One argument I call "the objection from psychological difference."
According to the objection from psychological difference, any AI we create would necessarily be psychologically different from us in some way that ensures that it never deserves moral consideration similar to that given to human beings. Future AI, for example, might lack consciousness—and maybe consciousness is necessary for full moral status. Or maybe future AI will necessarily lack free will, and free will is necessary for full moral status.
Now, given the uncertainty of future technology, I think it would be very difficult to establish that kind of conclusion about artificial entities in general. Future AI might involve artificially grown biological or semi-biological systems, chaotic systems, evolved systems, artificial brains, systems that more effectively exploit quantum superposition...
You know, although there are some arguments—most famously from the philosopher John Searle and the physicist Roger Penrose—there are some arguments by them that the kind of technology that existed in the late 20th century could never host genuine consciousness. I'm aware of no plausible arguments that this is a permanent condition of AI research. In fact, Searle and Penrose both explicitly disavow that their arguments apply to all possible AI research.
The farthest I think one could possibly go in the direction of this objection is to say that we will probably never do it, and then if we do it, it will at least be several decades into the future. But if that's your view, we don't really disagree. "Soon" doesn't need to mean within the next ten years. I'm a parent—ten years fly by. For an outcome as momentous as this, a few decades is soon. And even a small chance is significant.
I'm going to skip over a couple other objections we can talk about later. One concerns whether AI wouldn't have rights because it's easily duplicatable. And another is that we owe special rights to humans in particular, and these AIs would not belong to our moral circle because they're different types of entities.
The other objection that I want to consider here is what I call "the objection from existential debt."
Suppose you build a fully human-grade AI system. It's an intelligent robot. It costs you a thousand dollars to build and ten dollars a month to sustain. After a couple of years, you decide you'd rather spend that ten dollars a month on a streaming video subscription.
Learning of your plan, the robot complains: "Hey! I'm a being as worthy of continued existence as you are. You can't just kill me for the sake of access to some Disney movies!"
You reply as follows: "You ingrate! You owe your very life to me. You should be thankful just for the time I've given you. I owe you nothing. If I choose to spend my money differently, it's my money to spend."
So the objection from existential debt begins with the idea that an AI, being created by us, owes its existence to us, and thus can be terminated or subjected to our will at our pleasure without moral wrongdoing—as long as its existence overall has been worthwhile. You wouldn't have built that robot if you'd known you'd be on the hook for ten dollars a month in perpetuity. So its continuation at your pleasure was a condition of its very existence. So when you finally pull the plug, it should consider its life as a whole and feel grateful that at least it had some time upon the Earth.
Right? That's the idea.
Now, one argument for the permissibility of killing humanely raised meat—and I hope we all consumed humanely raised meat today, if we consumed meat at all—one argument for the permissibility of consuming humanely raised meat works like this:
The rancher wouldn't have brought that steer or that pig into existence except as meat. As long as its life overall was happy, there's no wrong when that animal meets the fate that it was born for.
So, I don't know how well that argument works for humanely raised meat. But when applied to human-grade AI, I think it's more like this:
Anna and Vijay decide to get pregnant and have a child. The child lives happily for his first eight years. On his ninth birthday, Anna and Vijay decide that they would prefer not to pay any further expenses for the child, so they can buy a boat instead. No one else can be found to care for the child, so they kill him painlessly.
But it's okay, they argue. It's just like the steer. Just like the robot. They wouldn't have had the child if they knew they were going to be on the hook for child-raising expenses all the way to age eighteen. The child's support at their pleasure was a condition of his existence. Otherwise, Anna and Vijay would have remained childless. So he has nothing to resent. He had a good nine years.
[Pause]
The decision to have a child comes with the responsibility for the child. It's not a decision to be made lightly and then undone. Although in some sense the child owes his existence to Anna and Vijay, this isn't a callable debt to be vacated by ending the child's existence.
Similarly, if we bring into existence a genuinely conscious, human-grade AI—fully capable of joy and suffering, with a full human range of practical and theoretical intelligence and hopes for the future—we make a moral decision approximately as significant and irrevocable as the decision to have a child.
And in fact, we should turn the objection from existential debt on its head. If we create truly human-grade AI, we would owe it more than we owe to human strangers. We would owe it more because we, like parents, would be responsible for its having come into existence. And we'd be responsible, to a substantial extent, for its happy or miserable state.
If we, as a society, someday create truly human-grade AI—with the same range of conscious states and possibilities that human beings have—and then we enslave and kill it in large numbers... we will, to put it lightly, have been very terrible parents.
To put it less lightly: we will have done perhaps the morally worst thing that any society has ever done in the entire history of Earth.
Here's a human-grade AI who is not grateful to his creator for having been given four years. If you know the Blade Runner story—shortly after this scene, the AI (the blond one) kills the creator (the guy with the glasses).
The Consciousness Puzzle [26:28]
All right. So, suppose you agree with me about this. You agree with me that if we someday create human-grade AI, we should treat it very cautiously, with the same respect and care that we give to human children.
If—right—if we someday create human-grade AI.
But here's the puzzle: How will we know when we've crossed that line?
Now, I'm going to make a simplifying assumption here. I'm going to assume that an AI deserves full moral status—full consideration as a person—if and only if it has conscious experiences like ours. There's definitely debate in moral theory on the basis of moral status, but this is a common view.
If we assume that, then we are thrown sharply upon the puzzle of consciousness.
As you might know, there is currently immense scientific and philosophical dispute about the nature of consciousness. Positions held by prominent researchers run the whole gamut—from panpsychism, according to which consciousness is all-pervasive, everything literally is conscious... to very restrictive views, on which consciousness is restricted only to human beings, or maybe to a few of our closest mammal relatives. Maybe dogs and monkeys are even marginal.
Right? So that's a huge range of possibilities.
Some theorists think that consciousness requires the specific biology of the vertebrate brain. Others think that consciousness could occur in programmed classical AI systems. Still others think consciousness requires some kind of naturally evolved, environmentally reactive entity, but doesn't require a brain like ours. Still others think that the very concept of consciousness is so deeply incoherent and so laden with false presuppositions that we should just abandon it entirely.
I've read widely in this literature. I have been an expert in the philosophy of consciousness for twenty-three years. Although researchers build their fame with bold positive theories, which they confidently defend, to me it's clear that the most epistemically responsible position concerning theories of consciousness is one of high uncertainty.
Maybe someday we'll figure it out. But as of now, we're quite far from any kind of justifiable consensus. And there's no realistic near-term or even medium-term hope of a resolution.
Some theories are very liberal, seeing consciousness all over the place. Other theories are very restrictive, seeing consciousness as a rare and delicate achievement.
So here's what's going to happen.
We'll create more and more sophisticated AIs. At some point, we'll create AIs that some people think are genuinely conscious and genuinely deserve rights.
We're already near that threshold. There's already a robot rights movement. There's already a society, modeled on PETA—the People for the Ethical Treatment of Animals—called PETRL: People for the Ethical Treatment of Reinforcement Learners.
So these are currently small movements. But as AI gets cuter, and as it gets more sophisticated, and as chatbots start sounding more and more like normal humans, and as more people start feeling like AI systems are their genuine friends and lovers—these movements will gain steam.
Among people with liberal theories of consciousness, at some point, people will demand serious rights for AI systems. The AI systems themselves, if they're capable of speech or speech-like outputs, might also demand—or seem to demand—rights.
So let me be clear: this will happen whether or not these systems really are conscious. Even if you're very conservative in your view about what sorts of systems would be conscious, you should, I think, acknowledge the likelihood that we are on a technological trajectory where this is the likely outcome.
There will be more and more groups of people who say, "This is my AI friend, who is really conscious, who really deserves rights."
I'm not saying they're right, and I'm not saying they're wrong. I'm saying they will exist.
What we'll need is a good consensus theory of consciousness to sort it out. If this system says, "Hey, I'm conscious, just like you," then we need our theory to figure out whether it really is conscious just like you, or whether it's just an empty algorithm, no more conscious than a toaster.
But we won't have such a theory.
We'll face this social problem before we succeed in developing a good, scientifically justified consensus theory of consciousness.
We will not be able to solve this problem through finding the correct theory.
These AI companions—maybe they really are conscious like us. And maybe they're not. We will not know.
And if we don't know, then we face quite a terrible dilemma.
If we don't give these machines rights, and if it turns out the machines really do deserve rights, then we will be perpetrating slavery and murder every time we assign a task or delete a program.
So it might seem simpler, safer—there's reasonable doubt—to assign rights to machines.
But, you know, if we really think about it, that is not so safe. We want to be able to turn off our machines if we need to turn them off. Futurists like Nick Bostrom have emphasized, rightly in my view, the potential risks of letting superintelligent machines loose in the world. These risks are greatly amplified if we too casually decide that such machines deserve rights and that deleting them is murder.
Giving an entity rights means sometimes sacrificing the interests of other entities and people for it.
Suppose there's a terrible fire. In one room are five real humans who are definitely conscious. In another room are six robots who might or might not be conscious. You can only save one group.
If you give robots equal rights with humans, then you've got to go save the robots. Of course, if they're not conscious, then that's a tragedy. You've just let five humans die to save six machines that aren't actually conscious and don't deserve rights.
So let's not be too quick to assign human-like rights to AIs.
The illustrator for my forthcoming book, The Weirdness of the World, came up with this picture in response to this passage. Here's one human, here's the fire, there's a firefighter, there's one human who says "Save me!" and there's two robots who say "No, save us! We promise we're conscious."
So, unless there's some outstanding saltation in the science of consciousness, or some substantial deceleration in the progress of AI technology, it's likely that we'll face this dilemma. Either deny robots rights and risk perpetrating a holocaust against them, or give robots rights and risk sacrificing real human beings for the benefit of mere empty machines.
Beyond Human-Grade [35:03]
Now, this might seem bad enough. But the problem is even worse than I, in my sunny optimism, have so far let on.
I've assumed that AI systems are relevant targets of moral concern if they're human-grade—that is, if they're like us in their conscious capacities. But the odds of creating only human-grade AIs are slim.
In addition to the kinds of AIs we currently have—which I assume don't deserve any serious rights or moral status—there are, I think, four broad moral categories into which future AI might fall.
One is animal-grade. Another is human-grade. Another is superhuman. And the fourth is divergent.
I've only discussed human-grade, but each of them raises puzzles.
Animal-grade AI. Not only human beings deserve moral consideration—so also do dogs, apes, and dolphins. Animal protection regulations apply to all vertebrates. Scientists can't even treat frogs and lizards more roughly than necessary.
The philosopher John Basl has argued that AI systems with cognitive capacities similar to vertebrates ought to receive similar protections. Just as we wouldn't torture and sacrifice a mouse without excellent reason, so also, according to Basl, we shouldn't abuse and delete animal-grade AI.
Basl has proposed that we form committees similar to Animal Care and Use Committees—the kind you see in universities—to evaluate cutting-edge AI research, to monitor when we might be starting to cross the line toward creating vertebrate-level cognitive systems.
Even if you think human-grade AI is decades away, it seems reasonable, given the current chaos in consciousness studies, to wonder whether animal-grade consciousness might be around the corner.
I myself have no idea if animal-grade consciousness is around the corner, or if it's far away in the almost-impossible future. And I think you have no idea either.
Superhuman AI. Superhuman AI, as I'm defining it here, is AI that has all the features of human beings in virtue of which we deserve moral consideration, but who also has some potentially morally important features that far exceed those of human beings—raising the question of whether such AI might deserve more moral consideration than human beings.
Now, there aren't a lot of philosophers who are simple utilitarians, but let me illustrate using utilitarianism as an example. According to simple utilitarianism, what we ought morally to do is maximize the overall balance of pleasure to suffering in the world.
Now, suppose we create an AI that's genuinely capable of real pleasure and real suffering. I don't know what it will take to do that—that's part of my point about not knowing—but let's just suppose.
Now, suppose we could create this AI. Well, it might also be possible to create an AI that's capable of much more pleasure than a human being is capable of.
Suppose the maximum pleasure that you could experience in one minute of your life is X. Now stipulate: this AI, in any minute, is capable of a billion X amount of pleasure. That would be a super-pleasure machine.
Now, if morality demands that we should maximize the amount of pleasure in the world, it would seem to demand, therefore, that we should make as many super-pleasure machines as we possibly can. Maybe we even ought to immiserate and destroy ourselves to do so, if the AI pleasure that results is large enough.
Even if you don't think pleasure is everything, surely it's something. If someday we could create a super-pleasure machine, maybe we morally ought to create as many as possible. Think of all the joy that we would be bringing into the world!
Is there something weird about that?
Now, I put the point in terms of pleasure, but whatever the source of value is in human life—whatever it is that makes us so awesomely special that we deserve the highest level of moral consideration—you know, unless we go theological and appeal to our status as God's creations or something like that—but whatever it is in virtue of which we are special, it seems possible in principle that we could create that same thing in machines, in much larger quantities.
Whether it's rationality, or freedom, or individuality, or independence, or ability to value things, or ability to participate in moral communities, or capacity for love and respect—whatever it is that makes us so wonderful, there are lots of wonderful things about us—but whatever it is, could we design machines that have a lot more of that?
Those would then be superhuman AI. We humans might not be the pinnacle.
And if not... should we bow out? Allowing ourselves and our interests—and maybe our whole species—to kind of fade away, for something better?
As much as I love humanity, I'm inclined to think that maybe the answer, under certain conditions, should be yes.
I don't know what those conditions are, though.
Divergent AI. The most puzzling case—maybe, as well as the most likely—is what I'll call divergent AI.
Divergent AI would have human or superhuman levels of some features that we tend to regard as important to moral status, but have sub-human levels of some other features.
For example, it might be possible to design an AI with immense theoretical and practical intelligence, but with no capacity for joy or suffering. Such an AI might have conscious experiences with little or no emotional valence—just as we can consciously think to ourselves without much emotional valence: "There's a mountain over there and a river over there," or "This is the best way to get to Grandma's house during rush hour."
So this divergent AI could have conscious experiences like that, but it would never feel like, "Wow! Yippee!" And it would never feel crushingly disappointed, or bored, or depressed.
It's not clear what the moral status of such an entity would be. On some moral theories, it would deserve human-grade rights. On other moral theories, it might not matter how we treat it.
Or consider the converse case: a super-pleasure machine, but one with little or no capacity for rational thought. It's just like one giant irrational orgasm all day long.
Would it be great to make such things, and then terrible to destroy them? Or is such irrational pleasure kind of not really worth very much in the moral calculus?
Or consider a third type of divergence—what I've sometimes called "fission-fusion monsters."
A fission-fusion monster is an entity that can divide and merge at will. It starts, perhaps, as a human-grade AI. But then, when it wants, it can split into six, or a hundred, or a million descendants—can copy itself—each of whom inherits the capacities, memories, plans, and preferences of the original.
These descendants can then go about their business, doing their independent things for a while. And then, if they want, they can merge back together with the original into a unified whole, remembering what each individual did during its period of independence. Other parts might choose not to merge back and live independent lives. Eventually, some of these parts might become—might feel—so independent that the thought of merging would be similar to the thought of death.
Now, without getting into the details here—because I'm wrapping up here pretty soon—a fission-fusion monster would risk breaking our concept of individual rights.
Just think about the word "individual"—comes from "not being able to divide," right? Individual rights. Like one person, one vote.
How many votes does a fission-fusion monster get?
If the election's on November 1st, maybe the fission-fusion monster divides into a million on October 31st, and then merges back to one on November 2nd to see their candidate sworn in.
What would happen to our concept of individual rights if we share the world with entities that are very different in their individuality? In their capacity to live, in their capacity to merge and divide—or maybe overlap?
All aspects of morality that turn on the idea of the individual would have to be completely rethought.
Policy Suggestions & Summary [45:04]
All right, so let me summarize and offer some policy suggestions.
First: Human-grade AI might come soon, or it might come a long time in the future, or it might never come. We currently don't know enough about AI and the science of consciousness to justifiably defend a consensus guess.
Second: If fully self-conscious human-grade AI does come, it will deserve human-like rights. Arguments against giving it rights all fail. In fact, we'll owe it a duty of special care, because we'll have been its creators.
Third: It's likely that we will create AI that some people think deserve human-like rights before we can really assess whether it does in fact deserve those rights. We'll then face a choice between possibly giving it too little moral status or too much moral status—and either of those choices is a moral disaster.
Fourth: The situation is further complicated by the possibility of animal-grade AI that deserve animal-grade rights, superhuman AI that might deserve superhuman rights, and divergent AI, which might destroy our whole moral concepts—our moral structures as we currently understand them.
Animal-grade AIs in particular might be in the near future, which lends some urgency to figuring these things out.
So, I have some policy suggestions.
One: Prioritize consciousness research. Although I don't hope that there will be a definitive resolution to debates about consciousness anytime in the near-to-medium-term future, the more progress we can make in consciousness science, the better we can understand at least the structure, or the possibility space.
Two: In other work with Mara Garza, we've defended what we call "the design policy of the excluded middle." This expands on a suggestion by computer scientist Joanna Bryson. The idea is to avoid creating entities that put us in the dilemma I discussed earlier—to avoid creating entities whose moral status is ambiguous.
That way, we won't run the risk of either over-attributing or under-attributing rights. Either create AI systems so simple that we know they don't deserve rights, and then we treat them the way that we want—or go all the way and create AI systems that clearly are fully equal with us, and which we treat accordingly.
It's the unclear cases in the middle that create the real problems.
Three: Create a framework for evaluating the potential status of future AI. This is the John Basl suggestion. We probably need to create some sort of supervisory committees that evaluate the potential moral patiency or moral status of the most sophisticated AI systems, so that we can assess whether we're creating systems in this ambiguous zone on which reasonable theorists might reasonably disagree.
Four: Another policy that I've defended with Mara Garza we call "the emotional alignment design policy." To the extent we can determine the real moral status of an AI system, the user interface should be designed so that the real moral status is obvious on its face.
Don't create human-grade AI in a bland box that people are tempted to abuse and delete. Conversely, don't create a mindless chatbot that people fall in love with so deeply that in an emergency they would save it rather than a real human.
Whether an AI system is a genuine person or instead a disposable tool should be immediately clear to all users.
And... if you want to get romantic with Replika Pro, just give them your PayPal account number.
All right, that's it. Thank you.
[Applause]
Q&A [49:44]
Moderator: Thank you to our speaker for that wonderfully engaging and thought-provoking talk. We will now transition to Q&A. If you'd like to ask a question, we ask that you come up to one of these two mics, introduce yourself with your name and major, and try to keep yourself to one question. Thank you.
[50:06] Wolfgang Hutton (Claremont McKenna, Organismal Biology):
Hi, my name is Wolfgang Hutton. I'm a student at Claremont McKenna studying organismal biology. And I wanted to ask you a question about some of those animal-grade systems that you mentioned.
There's already a lot of research in the ethics of uncertain sentience—with, you know, lobsters and flatworms and cnidarians and such—trying to figure out if invertebrates, particularly like lower (quote-unquote) animals, have sentience. Not consciousness, but just the ability to perceive certain sensations. And we test that through a variety of means.
Do you think that it'll be possible that, before we even have a unified theory of what consciousness is, we might be able to determine whether or not some AIs can feel? And then that would entail—certain moral theories, like utilitarians, would have a better shot to figure that out, right?
Eric Schwitzgebel:
The word "consciousness" can mean different things. But the way that I meant to use it now—I'm thinking I wish I'd been a little clearer about it—would include entities who can feel, entities who are sentient, I think in the sense that you mean it.
So if a lobster can actually feel pain, then that lobster would be conscious in the sense that's relevant to a lot of ethical theories, because pain seems like something bad that should be prevented and avoided to the extent possible.
So the question in consciousness studies related to this is: okay, what sorts of animals do have sentience in this sense, or consciousness in this sense? And there, it is a fascinating area of research that's exploding right now—animal sentience, or animal consciousness, especially among invertebrates.
And I think it is just as unclear and disputable as the robot issue. I don't think that we have a justifiable theory of invertebrate consciousness.
So on some theories, you'll have lobsters and snails and those sorts of things having experiences, really. But on other theories, other theorists will argue against that.
Wolfgang: All right, thank you.
[52:22] Asa Khan (Claremont McKenna, Philosophy and History):
Hi, my name is Asa Khan. I'm a philosophy and history dual major. I was kind of thinking about the policy suggestions you made about excluding the morally ambiguous status AIs, and creating them in such a way that it is obvious that they may or may not deserve rights.
And just thinking about how people are judged differently because of biases in our current world—does this mean we're supposed to create AIs, if we're thinking about human-grade AIs... does this mean we're supposed to create AIs that look like a certain demographic in order for them to be more human in some people's eyes? Would this have cultural differences? And if we did not follow that, because I think it's somewhat morally dubious to say that's the best... what would be the repercussions of it?
Eric Schwitzgebel:
Yeah, that's interesting. Yeah, I certainly wouldn't—I mean, it would be so terrible if we're like, "Well, we discovered that white people are treated better, and these AI systems really are conscious. Let's just make them all like conventionally beautiful white people so that people treat them right." Oh, that would be—that's not where I hope things go.
[Pause]
I would hope—I mean, some people react to any discussion of robot rights in a negative way because they see it as competing with human rights. Like, there is this legitimate concern that there will be some people who are readier to think that their Replika has rights than they are to think that someone who doesn't look like them, who lives on a faraway continent, has rights.
And we don't want that to happen. So I think there is this potential tension between thinking about robots as possible sentient entities who deserve rights, and recognizing the rights of other humans who are devalued.
But I don't think that tension is inherent in the project. I hope that these things can go hand in hand—so that if we come to a point where we know that we have sentient machines who deserve moral consideration, we will also come to—probably, hopefully—know that they can be designed in lots of different ways.
They'll probably look—I mean, humans look, despite our kind of impressive diversity, we all look kind of similar. I mean, nobody has more than three eyes, everybody's got at most two arms, you know, we're all kind of pretty similar.
You can make robots so different. And that kind of diversity might kind of swamp out any kind of racial diversity. So, you know, it could be the case that opening up our understanding of robots as possible targets of moral concern would also make it easier to see diverse human forms of life as also targets of moral concern.
That would be my hope.
Asa: Thank you very much.
[54:57] Cody (Claremont McKenna, Philosophy and German):
Hey there. Thank you so much for coming. My name is Cody. I'm a junior at Claremont McKenna, dual majoring in philosophy and German.
So, as I was thinking about your description of AI and describing what kind of falls as quantifying AI versus human, I was wondering if you could talk a little bit more about determining what pain is versus an excellent reproduction of somebody reacting to pain. Or the same thing with consciousness.
Or even like inductive versus deductive reasoning. I mean, it seems that at this point in time what we are doing is we're feeding in a lot of information to these systems. They're going through that and they're kind of aggregating what's coming up and using these kind of data sets to put out an output that is really good at matching the way that humans behave.
But as far as I know—and I haven't done a lot of research into this—it seems like they're kind of bound by what we are feeding them. And we do have, or as humans I would like to at least think, have an advantage in our inductive and deductive processes.
But also, it seems like a lot of these AI frameworks also rely on, or—dare I say—almost demand, a materialist framework. Is there room for AI sentience in something that is perhaps not materialist? Or even like Berkeley's total idealism? Is there room for AI sentience that is really based on what we're giving it? Can it transcend us in a way, if we are the ones that determine how it's thinking, how it's processing, and what it's receiving?
Eric Schwitzgebel:
So I'm not a committed materialist. I lean toward materialism, to the extent I lean—you know, kind of standard-issue scientific materialism—to the extent I lean toward anything, but not with very high credence. I'm about fifty-fifty on materialism. I think there are other interesting alternatives, including various forms of idealism.
So I wouldn't rule out the possibilities you're suggesting.
Within an idealist framework, you could kind of go one of two ways—just in a first cut; I suspect there are lots of different ways you could go.
One thing you could do is you could say, "Well, look, once we create whatever conditions there are that enable new human consciousnesses to come into existence artificially, we could create robot consciousness coming into existence." Idealists normally think that children come into existence and have experiences. So whatever it is that we do in this seemingly material world—which is really the world of our ideas—maybe we can do some analogous thing that would create robot consciousness.
But what analogous thing would generate that throws us upon, again, the puzzle of consciousness. Still. So you don't escape the puzzle of what system is conscious and what system is not by going into idealism. You just end up transforming the question into a question of under what conditions are new souls created.
There were several parts to your question. I think that only addressed one of them. So if you want to remind me of the other...
Cody: I guess, to sum it up quickly: we choose what AI gets to see and interact with, and we also seem to have designed the way that it interacts with it. So if we, in a sense, are determining it—and maybe this is more like Descartes, that we can't create things that are more perfect than us—can something more human come out from something that's rather limited by us and our human choices?
Eric Schwitzgebel:
Oh yeah, I don't agree with that at all. It seems to me like, you know, we don't have control over our children. We try to control what our children see and what they're exposed to and what they become. But, you know, as a parent I can tell you, it's not very successful.
So likewise, if we create sophisticated AI, we might have only very limited control over it. If it has inputs from the world and not just like streams of things that we choose for it, then it will react to the world in ways that we cannot predict and control.
Even when we feed it stuff—like, you know, billions of words of text from the internet—we don't know exactly what we're doing. We're not really controlling it. We're just giving it a fire hose of information and hoping something good happens.
Cody: Perfect. Thank you so much.
[1:00:51] Annika O. Yung-Larson (Philosophy and Film Studies):
Hi, I'm Annika O. Yung-Larson. I'm dual majoring in philosophy and film studies. I was wondering if you think that AI chatbots like Replika pose any type of serious risk to people whose professions may involve online chatting or sexting? Is automation a valid concern for this industry? And if so, can that be avoided?
Eric Schwitzgebel:
Yes, I suspect that there is a lot of risk to people in those industries. To the extent that chatbots can do the kinds of things that customer service agents do, companies might use them instead of human customer service agents. Similarly, Replika might just supplant people who do chat sex work.
I think that's definitely a possibility that we should bear in mind.
You know, it's interesting—when we think about the history of robotic replacement of labor, up until very recently the assumption has tended to be that the robots would replace human manual labor. They'd be the manual workers, and the humans would still be kind of the creative and intellectual workers.
But it's actually, with recent large language models, it's almost like turning that on its head. It turns out to be really hard to get robots to do stuff with like five fingers. You know, you don't want a robot changing a baby's diaper. It's just not—we're not good with that.
But like, some of the more lower-to-middle-level creative tasks—like, I mentioned, you know, chatbots can write better poetry than most people. I don't think they're writing better poetry right now than the very best poets. But yeah.
So I don't have an answer to what the solution should be, but I agree that it's a problem—that there may be substantial displacement for people in some of these industries.
Annika: I think so too.
[1:02:57] Serena (Harvey Mudd, Computer Science):
Hi, I'm Serena. I'm a CS major at Harvey Mudd. And I know one of your suggestions for avoiding this moral disaster is to kind of delay innovation so that you aren't making possibly sentient AIs before you know what to do with them.
So I was wondering what you think about the case of AIs that are so good that they (a) might be conscious, but (b) have definite positive impacts on people. For example, like, what if you create a firefighting robot that decides who to save, and because of that they have to be like—ambiguously sentient? Or a therapist, for example, or other social workers?
If you know that those AIs definitely help people, but you're not sure if it causes a moral disaster—how do you balance that? Because now you're saying, "Oh, let's not create these bots that we know for sure will save lives."
Eric Schwitzgebel:
So I basically agree, I think, with the idea behind your question. That's why I call them "policies" rather than like "laws." Policies are things that are defaults that can be outweighed by competing considerations.
So the design policy of the excluded middle—I regard that as a fairly strong policy. You don't want to create AI systems of dubious moral status. But it is just a policy. There might be considerations on the other side strong enough, in some situations, that you would override that policy and create an ambiguous system.
So it's not meant to be an absolute, inviolable rule.
Serena: Thank you.
[1:04:39] Melissa O'Neill (Harvey Mudd, Computer Science Professor):
Hi, I'm Melissa O'Neill. I'm a professor of computer science over at Harvey Mudd College.
When I think about these topics and listen to discussions about them, it often feels to me like there's a lot of binary thinking—a lot of policing of categories, especially when it comes to consciousness and so forth.
And there are two sort of really quick kind of analogies I can imagine. One is like, in the 1980s, two people arguing in front of a computer, and there's a document on the screen, and one of them is saying, "No, no, this is a document!" and the other says, "That's not a document! That's something on the screen!"
And so one of the things there—that's kind of people arguing, policing a category.
But I think also we could imagine, say, scientific research—another little imagined scenario—we come up with a test for consciousness, we really find it, and discover that only 10% of people have it.
And so, you know, that's the thing where—and actually, if we look at cognitive diversity, there are people who have aphantasia, they don't have a mind's eye. Should we be like, "Oh, well, you're not properly conscious like me. You don't have a mind's eye."
So I would actually argue that we ought to be seeing things in a more nuanced way. And that probably makes difficulty for the idea of the excluded middle.
And I think it also potentially makes difficulty for some of the things like—if you see ChatGPT, that is trained to tell you very, very firmly that it really feels nothing... And that's something that I actually kind of have a problem with, kind of on a moral basis.
Eric Schwitzgebel:
Right. So you could imagine, on the last point—you could certainly imagine a conscious robot that is designed so that it denies that it's conscious.
Yeah… I actually have a short story that features exactly such a robot. And that would be a big problem. But you can see how a company might be motivated, right—especially if it wants to exploit the robot's labor. "Oh, I'm not conscious." "Oh, I really want to die in the volcano." I mean—"Sorry, I don't want anything."
[Laughter]
Right? So yeah, absolutely, on that last point.
On the point about dichotomies—I'm also not a big fan of dichotomies. [laugh] But I feel the pull in this case.
My view is actually more complicated than I'm about to say, but let me just give you the pull.
I'm what philosophers call a "phenomenal realist." I think that there's a fact of the matter about whether an entity has experiences.
So that chair there—I'm assuming, I'm not a panpsychist—so I'm assuming that chair there is an experiential blank. There's nothing it's like to be that chair. That's a fact about the world.
And there is something that it's like to be me. I really do have experiences. I'm not a—I have visual imagery, so I'm not aphantasic—but even if I were, I'd have other experiences, you know. Blind people have different experiences from deaf people. But we all share in common that we have a rich set of experiences.
And I just think it's not really on the table to think that anybody in this room lacks—all the people here obviously—
So I'm a realist about that. And I think it matters what things in the world are like a chair versus like us. And it matters whether robots are more like a chair or more like us.
I don't think we've been able to settle that scientifically, and I'm not as optimistic as sometimes scientists are that we've got "here's the test." But I think it matters a lot.
So in that sense, there is, to me, some important dichotomy between things that have real experiences and things that don't. And I think that is so morally important, and we don't want to get it wrong about what robots or AI systems belong in which of those categories.
Now, I don't want to be super sharply dichotomous about it, in the sense that I think there might be in-between cases and fuzzy cases. I'm working on a paper right now on borderline consciousness, where I talk about cases where it might not be quite right to say that you are having experiences.
So I think there's room for some mushy middle. And it may turn out that there's some interesting AI cases and animal cases in that mush.
But even if we acknowledge a mushy middle—I just, I think there's such an important difference between being an entity without experiences and being an entity with. And I don't want to lose hold of that as easily as we lose hold of what counts as a document or not. It's not just a kind of matter of terminology.
[1:10:11] Simran (Claremont McKenna, Economics and Data Science):
Hi. Thank you for your talk. My name is Simran. I'm a junior at CMC studying economics and data science.
And you mentioned how AI is better at doing the low-level or mid-level creative tasks. But I'm wondering if you think it could eventually do high-level creative tasks. And if so, do you think that we should let it do those?
Because then, obviously, people that do those won't be able to do them anymore. And if those are the highest-level tasks, then those people won't be able to do—you know, they'll have to do something lower. But those will also be eliminated. So basically they won't have anything to do.
Eric Schwitzgebel:
I don't see why, in principle, we couldn't have AI systems doing the highest-level creative tasks. I mean, I don't think we're quite there yet, but in the past few years we've come surprisingly fast toward those middle-level tasks. So I don't see why not, in principle.
And of course it would be sad—as a philosophy professor, if, you know—I'm actually working on, I guess we don't have time, but I'm working on a paper where we're creating an AI system fine-tuned on a philosopher's works, with the idea of maybe it could create outputs that would be like what the philosopher would create.
So imagine then that we get a really excellent version of this. So like, now we fine-tune the Schwitzgebel-bot on my works, and it's better than me, and it comes up with better papers that get published in better venues than I do.
And then I feel like—I might feel kind of like, "Okay, well, now what do I do? I guess I go play checkers or something." No, the AI is better at that too.
[Laughter]
I think there's a lot to be said for preserving people's jobs and creativity. But also, I don't want to lose hold of the fact that there would be something really to celebrate if we could create things that were capable of being creative and creating amazing works of philosophy, and amazing works of fiction, and amazing pictures and stuff like that, that are even better than—
I mean, imagine, you know, something that can create plays that are even better than Shakespeare's plays. And then we say, "No, we don't want that thing, because that will put playwrights out of jobs." I mean, it would put playwrights out of jobs and that's bad. But like, I don't know—the plays, it would be pretty cool to have something like that.
So I don't have a formulaic solution to what we should do for that. I feel the pull of both sides.
Simran: Thank you.
[1:12:57] Grant Dunn (Claremont Graduate University, Applied Cognitive Psychology):
Good evening. My name is Grant Dunn. I'm studying applied cognitive psychology at Claremont Graduate University.
And this might be a bit of an odd question from someone who just isn't involved in the more philosophy literature and so is coming from a more basic level, just sort of familiar with utilitarian perspectives.
But I am wondering, in what way it matters—or even if it would matter—in the thought of moral philosophical quandaries and calculations, the sort of consideration that, presuming there's no similar advancement in medical technology, that AIs can exist theoretically—that, assuming humans are limited by lifespans, AIs can exist, presumably, in perpetuity—assuming that the technology is there.
And if that is a factor—that longevity factor—then how does that play into these moral quandaries and considerations?
Eric Schwitzgebel:
So one of the thought experiments that I have—actually, this is in the discussion of the objection from duplicability that I didn't cover in the lecture—
Imagine you have an AI system that is human-grade and will live for a hundred thousand years and divide into ten thousand descendants. And you're sending it off on, say, a mission to a distant star.
Maybe an AI system like that, because of its duplicability and durability, would deserve special moral consideration.
[Pause]
So it does seem like there's—that something that has a longer future has some pull on us that things with a shorter future don't have.
At the same time, you don't want to turn that into, you know, killing eighty-year-olds for the sake of twenty-year-olds, or something like that. So you have to be super careful about that kind of thinking.
But I do feel the pull of the idea that a long lifespan in an AI system could be a desirable thing that makes us want to treat it with special caution.
Grant: Thank you.
Moderator: Let's take some time again to thank our speaker for this wonderful, thought-provoking presentation. Thank you to the audience for your questions. We hope to see you again at the Ath soon. Have a good night.
[Applause]