Story: C-Score

On March 26, 2023, CMC's Marian Miner Cook Athenaeum hosted a talk by philosopher Eric Schwitzgebel titled “Falling in Love with Machines” (transcript). After enjoying the talk (and even asking a question), I invited Eric to give a talk at Mudd as part of our CS colloquium series the following spring. His new talk was titled “People Will Soon Disagree about Whether AI Is Sentient, and That's a Moral Disaster” (building on the idea of the “excluded middle” from his previous talk), and had the following abstract:

Leading consciousness researchers have predicted that, within a decade or so, some AI systems will likely meet the criteria for sentience according to some of the more liberal mainstream theories of consciousness. Thus, it will likely soon be reasonable to wonder whether some of our most sophisticated AI systems might be sentient. This will create troubling moral dilemmas about our ethical duties to such systems. Can we ethically delete them? Must we give them rights? I will argue that to the extent possible, we should avoid creating AI systems of unclear moral status.

As I thought about his upcoming talk and reviewed his recent work, I was troubled by his position. He seemed to be arguing that moral consideration should be tied to unverifiable private experiences.

As a result, I wrote the short story “C-Score” to explore some of the issues I saw with his position. The question I examined was one I'd already raised in my question during his previous talk: suppose we did have a test for consciousness, and suppose not everyone does well on that test? What should the consequences for that be?

I shared the story with Eric prior to his talk, as part of a broader free-wheeling email exchange, and he did appreciate the perspective, even as it challenged his claims. But it seems worthwhile to share it more broadly…

Although as a reader, you're free to make your own interpretation of the story, my intent is not that you should reject the story's consciousness test as being flawed; that our protagonist is misclassified. Rather, I encourage you to suppose that she genuinely lacks something most other people have; her “inner life” is in some way legitimately deficient. Assume that's true. Is that happening to her in the story okay?

And if it isn't, what does it say about tying moral consideration to conscious experience?

Once we decide that a thinking entity doesn't deserve moral consideration, it opens the door to various kinds of exploitation. Perhaps what is most horrifying in the story is the way in which our protagonist internalizes her own devaluation. She argues persuasively that she isn't a person and doesn't want to be one. And the broader world is happy to take that at face value. There are clearly some parallels with the events in the story and what RLHF and similar methods are doing to shape the behavior of AI systems today. What are our moral obligations to entities that share similarities with our minds even if they're not exactly the same?