I wanted one of my first posts to be about the Chinese Room thought experiment, as it combines my interests in language, A.I., and consciousness. In this post, I’ll lay out the basic overview, and will have some more detailed follow-ups in later posts.
The Chinese Room thought experiment originated with John Searle in a 1980 paper. You can download the paper here (PDF warning), and read detailed overviews at the Stanford Encyclopedia of Philosophy, Wikipedia, and the Internet Encyclopedia of Philosophy.
In a nutshell, Searle is arguing against the idea of strong A.I., i.e. against the idea that a computer program (and therefore an artificial agent) could possess mind, consciousness, or understanding.
Imagine a person inside a room with very limited communication to the outside world. This person knows English but not Chinese. This person receives input sequences of Chinese characters from outside the room. The person possesses a book in the room that is basically a giant lookup table: it tells the person what output to generate, based on the inputs that are received. Once the person consults the book and figures out what output they should generate, they send a sequence of Chinese characters back to the outside world.
Now let’s consider this from the outside perspective. A person who knows Chinese could write a question in Mandarin, e.g. a question about where to find a good restaurant in Shanghai, input the question to the room, and get a relevant response (e.g. a recommendation). To that outsider, it seems very much like the person inside the room knows Chinese. They could carry on a coherent conversation indefinitely.
At this point, we’ve described the basic setup of the thought experiment. So what’s the point? The point is that the person inside the room does not actually know Chinese. They are just doing symbol manipulation by taking the inputs symbols, looking them up in the big book, and outputting whatever symbol sequence the book tells them to. And the workings of the Chinese Room are meant to be analogous to the workings of a computer, so Searle’s basic point is that a computer conversing in Chinese would similarly not really understand Chinese; it would just be doing symbol manipulation.
(Note: The term “strong A.I.” is a bit vague, and some would use it just to mean A.I. that performs at or above human-level on certain tasks. The Chinese Room thought experiment is not an argument against such superintelligent A.I. systems.)
We’ll find that this type of thought experiment is common in consciousness studies and philosophy of mind. It also bears some resemblance to the Mill Argument of Leibniz. I find the Chinese Room to be a very compelling thought experiment. It’s generated a huge amount of literature and critical responses in the ensuing decades. I’ll mention one of the critical responses very briefly here.
The Systems Reply: This reply essentially says that, while the person in the room does not understand Chinese, the overall system — consisting of the person and the book — does understand Chinese. An outsider interacting with the system would certainly feel that the system understands Chinese, because they could have a fluent conversation together. The word “understands” seems a bit vague here — we’ll come back to that shortly. Searle’s response to the Systems Reply is to create a modified version of the Chinese Room. Now imagine that instead of the person in the room having a book that they consult, they’ve instead memorized the entire book, through some amazing feat of memory. It’s still just a big lookup table, but it’s all in the person’s mind now. So the system works just as before — communicating with outsiders in Chinese — but now the system consists of only the person. Surely we don’t want to say that this system understands Chinese.
You might be feeling some hesitation now — if this person has memorized the entire book and can converse fluently in Chinese, surely we do want to say that they do understand Chinese, right? Hasn’t Searle undermined himself with this revised thought experiment? No — we need to revisit the notion of understanding now. Remember that the person has no idea what the inputs or outputs mean. He has no idea that an input was a request for restaurant recommendations in Shanghai, and no idea the output he generated was a recommendation for a particular restaurant. It is all just symbols to him. And if you talked to him later, in English, he’d tell you that he had no idea what any of it meant.
There are other responses to the Chinese Room that I’ll cover in upcoming posts, and I’ll talk more about the differences between syntax and semantics.
