Moderator: Heinz R. Pagels
Panel Members: Hubert L. Dreyfus, John Mccarthy, Marvin L. Minsky, Seymour Papert, and John Searle
Source: Pagels, H. R., Dreyfus, H. L., McCarthy, J., Minsky, M. L., Papert, S., & Searle, J. R. (1984). Has artificial intelligence research illuminated human thinking? In Computer Culture: The Scientific, Intellectual and Social Impact of the Computer (pp. 138-160). New York Academy of Sciences.
H. R. PAGELS: A few years ago, I asked a colleague of mine at Harvard what was the future of artificial intelligence. He simply said: “Heinz, if you took the smartest two dozen people of the eleventh century and put them in a room together and instructed them to put together a model of the physical universe, there’s no question that they would come out with something that would be absolutely brilliant. But it would be all wrong, because the concepts were not to be invented until several centuries in the future. That’s similar to the case for the AI proponents-they are very smart people, but the right concepts are not yet available.”
People who work in artificial intelligence research have to contend with critics of that kind, and I’m sure we’ll hear a good deal about that. I once asked Marvin Minsky why this field of study was ever called artificial intelligence. I said, “Why didn’t you call it something more general, like cognitive science?” And Marvin responded: “If we ever called it anything other than artificial intelligence, we wouldn’t have gotten into the universities. Now that we’re in, and the philosophers and the psychologists know that we’re the enemy, it’s too late.”
Some of the topics we’re going to discuss here revolve around the issue of consciousness. There is a problem with regard to consciousness which can be stated very simply. I know that I’m conscious. I’m a thinking, experiencing being. I can close my eyes and think; I can dream; I know that I’m conscious. But as philosophers have shown in debate over the centuries, there’s absolutely no way that I can prove to you that I am a conscious being. You wouldn’t know, for example, that I wasn’t a mindless machine put together molecule by molecule by an extraterrestrial civilization and sent here to confound you. Even if you knew what was going on in every single neuron in my brain, you could not prove that I was a conscious being.
Some people on this panel may contest that, but the real question is, can we determine whether or not machines are conscious? Many decades ago, Alan Turing addressed that question and came up with the famous Turing test, a purely behavioral model, and I’m sure some of the panelists here will allude to that test.
I want to emphasize that artificial intelligence does not have to do with the technology-with the microchips and the miracles being performed by electronics engineers. Artificial intelligence as viewed by its proponents is really a philosophy, almost “the world view.” And in many ways, it’s competing with other, older world views and confronting philosophical problems that have been with us for a long time.
I want to point out that the speakers are not divided into two classes-those in favor of AI and those in opposition. The individuals who are here all differ among themselves. At this point each of our panelists will very briefly describe his position and outlook on artificial intelligence and the question before us: “Has AI research illuminated human thinking?” And then we will engage in an open discussion. Professor Minsky will begin.
M. MINSKY: I believe the question that we’re trying to discuss is, If you made a machine that looks as if it thinks, would it really think? And there are two parts to that. There’s the “if”-Can you make a machine that appears to be intelligent? We don’t know the answer to that of course, but I think it will be yes, and that after a long time, when we solve such problems as commonsense reasoning and representation of knowledge, the computer programs and machines that are built according to those principles will get smarter and smarter and more lifelike in some ways.
Then there are other issues about how similar the machines will be to humans and how much they’ll resemble us. There’s another question yet, which is, What does it feel like inside them? These issues are complicated. For example, one of the things that I’ve worked on recently is the theory of jokes and humor. Many people will say they can see how you could make a machine intelligent, but not how you could make it understand humor or fear. Well, those people are wrong. They can’t see how to make a machine intelligent, yet they think somehow that it’s easy and that it’s the emotional aspects that are difficult. This just isn’t true; it’s a major superstition of our culture that feeling an emotion is very deep and hard and difficult to understand, whereas intellect-how we get ideas, how we think is easy to understand. In that perspective, many of the issues that we’re about to discuss here will seem silly to me because they’re missing the point of what’s hard.
Say, somebody tells you he just got an idea. He could build a car with eight wheels and it would go over bumps easier. Instead of criticizing the idea, suppose you ask him how he got that idea. What will he say? He might say it just popped into his head or he just thought of it, or it came to him. Isn’t that shocking? Yet if he says he feels bad and you ask him why, then he’ll tell you something pretty interesting and simple-“Well this room I went into reminded me of the one that my sick friend was in.” He’ll tell you why he felt. I believe that’s why Freud worked on emotions, not because he thought they were deeper, but because he thought they were shallower.
So everybody’s got it wrong. It seems to me that we understand emotions rather well. But when it comes to ideas, my image is that people see themselves living in a world of thoughts with a brain that has an almost impermeable shield around it, but that every now and then some idea leaks through and gets in-and that’s about it. The problem with these people who say that you could make a machine think, but it wouldn’t really feel like us is that they don’t seem to have thought about the real problem. I’ll end with that.
Seymour Papert: The question that I would like to talk about is, has what’s been done in artificial intelligence led to deeper understanding of better ways to think about human intelligence? And by human intelligence I mean everything that happens in the human mind, in the broad sense to include jokes and feelings as much as reasoning.
First, I want to make the distinction that the question of whether artificial intelligence has illuminated human intelligence is very different from the question of how intelligent machines really are or ever will be, or whether they work the same as people do, and so on. It’s quite clear, for example, that one way to illuminate human intelligence is by contrast. In a sense it could well be that the more machines think in different ways from people, the more valuable they will be as means of illuminating our understanding of ourselves. To understand something, you want to know what else it’s like, but you also want to know what else it’s different from.
Clearly, until very recently, attempts to understand the human mind in terms of what it’s different from have been of two types: one is by comparison with animals, and the other by comparison with various mythical, invented beings such as gods and other mythical creatures. We now have a third point of comparison, namely, computers. And I do think that, at the very least, we have been led to be much more precise in a lot of the thinking about our own minds by having this new point of comparison, by being able to discuss whether we are like or unlike these machines, irrespective of whether the answer is yes or no.
I would push that a little further and say that one of the effects of artificial intelligence has been to introduce more structure into our thinking about human thinking. Because if you think of thinking only as one kind of thing, you can’t get your teeth into very much. To be specific, what I think is probably the best-developed, detailed attempt to make a theory of human thinking that actually matches the way people think is Newell and Simon’s attempts to simulate the solving of certain puzzles and the playing of chess. Well, is this a good model of how people think? There’s a terrible pun in that. I believe it’s pretty obvious what the answer is that sometimes people think like Newell and Simon describe and sometimes, most often, not. I don’t believe I’m thinking like that now, for example. But the fact that they have described this particular model of one particular way of thinking in a lot of detail means I can talk in a much more dense and technical way about other ways of thinking and so put more structure into comparative ways of thinking. This is the most fruitful way in which AI has illuminated human intelligence, by enabling us to be more specific about different ways in which the mind might work. Before AI, there was very little structured classification of ways of thinking.
I would like to push that in another direction. I take it that the question Has artificial intelligence illuminated the human intelligence? means Has AI acted as a searchlight for psychologists or other people to understand better how human intelligence works? But you could twist those same words around and ask whether it has made human intelligence work any better than it did before. I believe that this might be as much of a contribution of artificial intelligence to the world as the making of robots. I’ve seen some examples of this working with children. Giving children very simple AI models-extremely simple models of how the thought process works in certain situations-enables them to think more clearly about their own thinking and so be more critical about themselves and so be more constructive in taking the next step further. I choose and emphasize this example of children because I think it pushes to an extreme a remark I made at the beginning. These models of thinking that we give to these children have no pretention to be universal and complete models of how the mind works. They are only a little theoretical model that captures enough of what’s happening there for you to think about it.
Now ultimately all theoretical models are of that sort. By simplifying reality and pulling out some aspect of it that you really want to think about now, they enable you to think better about reality. And I do think that artificial intelligence has done this quite irrespective of whether we think that present machines are suitable models for the whole human intelligence (which they certainly aren’t), or even of whether we think they ever will be. That’s the main point I wanted to make to take some sort of position for discussion.
I would now like to make a brief comment on a point raised by Marvin Minsky about whether feelings are harder to understand than thinking. I’d like to make two remarks about that. The first is that, when he expressed such an unpopular position, it reminded me that we are all very inhibited and often embarrassed about expressing what we really think about our own minds and what goes on in them. I believe this very much colors discussion on these issues and that, very likely, the attempt to make explicit models breaks down these inhibitions. This is another way in which AI is helping, by provoking us to let people see more into how we think and lay it out more than we used to.
The second is that I don’t think emotions are harder to understand than ideas are but consider the following. If you made a machine that simulated intelligence, I think it’s obvious that we could argue about whether it is intelligent. And I think that most people would go along with the assumption that simulated intelligence is at least some kind of intelligence. I’m going to contrast this with the question that simulated feelings are in a very different state, that is, whereas most people would go along with the statement that simulated thinking is thinking, simulated reason is reason, most people would have serious doubts about whether to go along with the statement that simulated feelings are feelings. And yet I think this difference, which is a matter of house psychology, has very little to do with the substance of what machines can do and what they can’t do.
J. MCCARTHY: The question is whether AI has illuminated human intelligence, and I think the answer is obviously yes. AI and psychology influenced by AI are responsible for destroying behaviorism as a serious approach to psychology and turning psychologists toward information processing models. Presumably a psychologist would be more competent than an AI person to speak about that influence.
Now I want to deal with the issue about whether a machine really thinks or believes. This is an elaboration of a point I made in my paper. Namely, we will find it necessary to use mentalistic terminology in describing what we know about machines. Of course, if we understand how a thermostat works, we don’t have to adopt the mentalistic stance of saying that the thermostat thinks the room is too warm.
Indeed I picked the thermostat example precisely because we can understand it both ways-mechanistically and mentalistically. Just because we can understand its mechanism is not a reason to bar the use of mentalistic terms. There’s an illuminating analogy with the number system and its historical development. Suppose someone said that he didn’t think that one is a number, arguing that if you have only one thing you don’t have to count. Indeed, most languages treat one differently from the other numbers. Some treat two differently also, and in Russian numbers up to four take the genitive case. The introduction of zero to the number system is even more recent, and I believe it was controversial. The justification is that the number system as a system makes more sense if both zero and one are included. Likewise, a systematic treatment of belief by machines will have to start with the easy cases.
A more complex case arises when we say that a dog wants to go out. We cannot practically reduce this to a propensity to behave in a certain way, because we may not know what the dog will do to further this desire. It may scratch the door or yelp or whatever. Secondly, we may not know the evidence that the dog wants to go out. Therefore, the fact that the dog wants to go out is best treated as primary.
Another useful idea comes from Dan Dennett-the notion of the “design stance.” Suppose we are designing a dog as an artificial intelligence. It will be convenient to design in the desire to go out as a possible state. We have a variety of choices as to what will cause this state and what our dog will do to realize the desire. In designing computer systems, we will also find this notion of wanting a useful intermediate point.
As far as I can see, the purely intellectual terms are easier to handle for machines than are some of the emotional terms. “It believes” is easier than “it hopes,” which is easier than “it likes me” or “it doesn’t like me.” And as to whether the machine is suffering, all I can say is that it complains a lot.
When we ask whether it is conscious, there are a lot of criteria for saying no. No, because it doesn’t know about its physical body. No, it doesn’t even refer to itself. On the other hand, it might claim to be alienated, but it has just read Marcuse. Well, that’s how most people who claim to be alienated come to claim it. It’s something they read about.
H. L. DREYFUS: I want to respond to both these questions: Where are we in artificial intelligence? and What has the work in artificial intelligence taught us about thinking? I agree with Seymour Papert that it has taught us a lot, if in no other way than by contrast.
But first, I want to catch up on what was happening this morning, because I want to take off from what we heard about expert systems. As Marvin Minsky said, expert systems only work in microworlds. Microworlds have bracketed off all of commonsense knowledge. I think that’s a very important remark to enable us to see where the interesting and essential problems arise for an attempt to understand the mind on an information processing or computer model.
Now the successful expert systems are in a way like games. They operate in a circumscribed domain where what’s relevant or not relevant has been settled before the game starts. The easiest way to see that is to think about a game like chess, which is a microworld. It’s always relevant where each piece is placed, what kind of piece it is (whether it’s a rook or a bishop), and whether it’s white or black. It’s never relevant how heavy the piece is, what temperature it is, whether it’s fancy, carved, or plain, whether it’s in the middle of the square or on the side, whether it’s clean or dirty, etc. In the same way, DENDRAL, the program that is an expert in spectrograph analysis, deals only with spectral lines-a problem that relates to human beings can never arise. That is a completely circumscribed microworld or domain. Even a program like MYCIN, which looks like it deals with human beings, deals specifically only with objective scores on blood tests. It doesn’t connect up with everyday human activity.
When you do connect up with everyday human activity, when you try to make an expert system that captures the expertise of ordinary people, then you get out of microworlds and into the problem of what’s called commonsense knowledge. Commonsense knowledge deals not only with facts about the everyday world such as the physics of the everyday world, but also with exceptions-the ceteris paribus rules of the everyday world. That’s a fancy way of saying rules that say, “everything else being equal, then such and such.” Marvin Minsky gave a good example of that-as a rule, birds are animals that can fly; and when people talk about birds, you can assume that they’re talking about birds that fly. But if their wings are broken, as Minsky said, or if they are toys, or even if they’re penguins, then birds can’t fly.
A way to connect this with the talk this morning about expert systems is to use as an example the problems that arise when you try to automate a travel agent. The travel agent is an interesting area in which the microworld, or isolated domain, where expert systems have worked and will do important things shades over into the everyday world. You can give your automated travel agent lots of facts. You can tell it about airports, rates, and distances. You can also give it the kind of facts that expert travel agents know. Facts like, don’t try to go to another airline terminal when changing planes at an airport if you only have half an hour, etc.
But there’s something else which is harder and so will give you an idea of what I mean by commonsense knowledge. If for instance you say to this automated travel agent, “I’d like a flight to San Francisco leaving at around 6:00.” And it tells you that there’s a flight leaving Kennedy at 6:30. Then you say, “No, I’d like something a little bit earlier.” And it looks up its schedules and says, “There is a flight leaving at 6:29.” Something has gone wrong: 6:29 is a little bit earlier than 6:30 alright, but it’s not something any human being would want to know. The automated travel agent doesn’t contain any knowledge of human temporality-what spans of time are important to human beings.
Now you might think that you could just fix that. But it’s not so easy. You can’t just give it a rule like “a little bit earlier means at least 15 minutes earlier,” because that is a ceteris paribus rule, the kind of rule human beings could use if they were travel agents. But everything else often isn’t equal. Sometimes you really do need a flight ten minutes earlier if it’s a tight connection. And sometimes even an hour earlier might be okay, if you’re flying to Australia for example. So, what you need is some sense of the background of everyday needs and experience that human beings bring to the question, Do you have something a little bit earlier? And that means you not only have to put into your expert system all the facts about airports and airplanes, but something like the facts about what it is to be a human being. Because a computer hasn’t the slightest idea of what it is to be a human being. You have to tell it everything that there is to know about human beings if it’s going to have the kind of commonsense background knowledge that human beings have.
Now of course, it won’t need to know everything before it gets pretty good. Indeed, if you really tried to tell it everything, it would be an infinite task. There’s a general agreement between John McCarthy and Marvin Minsky that it might take, say, 300 years before you get commonsense knowledge into a machine. What they mean is that in 300 years we might get enough of the facts about human beings into the machine so it will begin to be able to behave intelligently, and we could then get expert systems out of microworlds into the real world and start capturing the kind of expertise that travel agents and bankers and literary agents have, which is always an expertise that opens out into the rest of human common sense.
That’s the first half of what I have to say. Now I want to talk about how human thinking differs from computer thinking, because the question before this panel comes down to, How do humans deal with common sense and is their way different from the way computers would deal with common sense?
My view is that if commonsense understanding were just knowledge, then programming it would only be an infinite task and we could get on with it and maybe we’d be somewhere with it in 300 years. But I think commonsense understanding isn’t a kind of knowledge at all. And that’s where I want to agree with Seymour Papert that we can learn a lot by contrasting computer “thinking” and human thinking. What we will learn by contrast is that people don’t have a lot of facts and rules in their minds for understanding the everyday world. They’ve got a kind of skill for coping with things. Therefore, as Minsky says, the right action usually just pops into their head. They’re able to see what the issues are and to see what’s relevant in a situation.
This ability that people have, to see what’s relevant and to have the right thing pop into their head, presupposes something like knowing what matters. But I don’t think computers as we now understand them have anything like mattering or concern. It also requires having images and having memories since it involves seeing the current situation as resembling earlier situations, where resembling is a tricky notion because resembling doesn’t mean identical with respect to any particular features, which is the way machines always have to analyze resemblance, but simply overall similarity.
Since human intelligence is not a matter of knowledge but a matter of understanding, AI is not even moving in the right dimension. To work at the commonsense knowledge problem is like trying to get to the moon by climbing higher and higher in a tree. You’re just not going to get there that way although you are getting a little bit closer.
Now we don’t need commonsense knowledge to be intelligent and know what matters to our kind of being because we are it. We don’t need to know about bodies because we are bodies, and the same for emotions and situations. That is, we are our bodies, we have emotions, and we’re in situations whereas computers are outside and have to be given knowledge of all that. That, I think, poses an insurmountable problem for AI as it’s now practiced and lets us see how our kind of thinking is totally different from the computer’s kind of thinking.
J. SEARLE: Well, I’m distressed to find that I agree with a lot that everybody said. However, I’m going to try to state some positions where I disagree with what I take to be certain common tendencies in artificial intelligence.
I want to remark that the question we’re supposed to be talking about got subtly rewritten. It says, Does AI illuminate human thinking? At some level, that means the kind of stuff that’s going on in us now. Not all of our thinking is conscious, of course, but some is and some of it’s unconscious. I want to focus a little bit on the question of what sorts of significance we should attach to artificial intelligence research into thought processes, conscious and unconscious.
In doing that, it seems to me we ought to make some distinctions. The computer is a terrific tool and there’s no question but that it’s going to be a useful tool in studying human beings and human thought processes just as it’s a terrific tool in studying fires or patterns of crime or the marginal propensity to consume or all sorts of other things. But when you read the AI literature, you discover something amazing. There are a lot of people who don’t think we can just use the computer to study understanding the way we can use the computer to study five-alarm fires and big rainstorms in California. Rather, it turns out that they think the appropriately programmed computer literally does think in exactly the same sense that you and I think-that it literally has the kind of mental processes that you and I are having. So, I want to distinguish what I call “weak AI,” which is the view that says, “Sure, use the computer; it’s a terrific tool,” from a much stronger view that says: “It isn’t just that we’re simulating thinking or studying thinking. Our appropriately programmed computers with the right inputs and outputs will literally have thought processes, conscious and otherwise, in the same sense that you and I do.” I call this view “strong AI.”
Now I like that thesis because it’s clear that we know exactly what somebody’s saying when he says, “Look, my machine or the machine we’re going to eventually build has thought processes in exactly the same sense that you and I have thought processes.” It’s clear, and it is false, and it is demonstrably false. I’m going to take a couple of minutes to demonstrate its falsity, so there won’t be any illusion that we’re all one big community of agreement.
The way I like to demonstrate the falsity of strong AI is to get you to imagine yourself instantiating a computer program for a certain kind of thought process. It’s very important in these discussions to take the first-person point of view, to ask, what would it be like for me? Because that’s what we know of being conscious and having thought processes. So, imagine that there’s a computer program for understanding Chinese, so that if you punch a question in Chinese into the computer, the computer can give out the right sort of answer. It has the right sort of data base and the right kind of program so that it can process questions in Chinese and give the right answers.
Now imagine that you are the computer. You’re locked in a room, and a lot of Chinese symbols are in the room together with a whole lot of rule books for shuffling these Chinese symbols around. This will only work if you don’t know Chinese. Like me; I don’t know a word of Chinese. I don’t know what any of these symbols mean. So, there I am in the room shuffling these symbols around. The questions come in. I look up what I’m supposed to do when I get a squiggle squiggle sign, and I go and match it with a squaggle squaggle sign. That is called a computational process over a purely formally specified element. These are what Simon and Newell called physical symbols; and I am now acting as a physical symbol system.
Let’s suppose these guys get good at writing the programs. I get good at shuffling the symbols. The questions come in, and I give out the right answers. One guy in responding to me said, “Suppose one of the questions is ‘Do you understand Chinese?’ ” And I shuffle around-now I don’t know what any of these symbols mean-and put out the symbol that says: “You bet I understand Chinese. And how! What could be more obvious? Why do you keep asking me these dumb questions?” What I want to say is that it’s quite obvious, once you look at it from the first-person point of view, that I don’t understand a word of Chinese, and I wouldn’t learn Chinese from instantiating the Chinese understanding program.
Why not? What is it that I have in English that the computer doesn’t have in Chinese? Notice that if I don’t understand Chinese in that story, then neither does any other computer program understand Chinese, because the program hasn’t got anything that I haven’t got in the story. What is it that I’ve got for English that the computer program doesn’t have? Well I like commonsense answers. The difference is that in English, I know the meanings of the words, and in Chinese, I don’t know the meanings of the words-all I’ve got is a set of formal symbols with a set of computational rules for manipulating the formal symbols.
The point of the parable about the Chinese room is to reveal a deep point about the character of artificial intelligence research and human thinking. This is the point I want to leave you with, that is, from syntax alone you can’t get semantics. From purely formal, symbolic operations, you can’t get the mental content. So what I’m trying to remind you of with this story about the Chinese room is something we know independently anyway, namely, what the computer has as a computer is a purely formal level of operation. That’s its great appeal. The same program can be put in a complete variety of different kinds of substances, in different kinds of hardware. But as far as we know anything about how the real world works, the world in which we live, our mental states have to be something much more than just a set of syntactical processes because we actually do have thoughts and feelings. We actually have mental contents.
Why is it that the computer doesn’t have those in the sense that we do? Let’s take the case of thirst, because I’m now thirsty. As far as we know anything about it, thirst is produced in the hypothalamus by the action of angiotensin, which is synthesized by the secretions of renin from the kidneys. The point is that there’s a quite specific story about how it works, and the result is that I now feel thirsty. Now think about what an AI program would do. An AI program would say, “Well look, there’s just a set of formal syntactical processes; that’s all there is to feeling thirsty.” The way to see that this can’t be right from what we know about how the world works is to imagine that we put the thirst feeling program in some completely different sort of system. Make your computer out of old beer cans, take millions of old cans and let them bang together to simulate the neuron firings of the synapses. Now notice what strong AI has to claim. It can’t be just claiming: “Well who knows, maybe the system of beer cans is thirsty. I mean do we know so much about what it’s like to be a beer can that we’re sure that the beer can isn’t thirsty?” That’s not the claim. The claim has to be that the system must be thirsty because all there is to being thirsty is instantiating the formal computer program for being thirsty. I want to say that if we know anything to be false, we know that to be false. We know it quite independently of these discussions because we know that from a formal level of symbol manipulation by itself, you don’t get semantic content.
Well, why do people get in this bind? I mean why would anybody want to maintain these views? There are a couple of reasons. One is the constant adoption of a third-person point of view. We’re always asked how would we know that some other system was thirsty or understood Chinese. That’s the whole point of the Turing test-take the third-person point of view. But I want to tell you something about the mind. At some point, you’ll only understand what a mind is and what it’s like to have a mind by having one, by being one, by adopting the first-person point of view.
So there is this objectivizing tendency. It’s part of modern life that we think all of knowledge must be described from a third-person point of view. But it’s obviously false, if you think about the character of our mental states.
The second feature-and this is a kind of linguistic problem that gets into these discussions-is that we find it completely natural to use mental terms in a metaphorical extended sense. It seems to me we do that all the time. I mean I apply all kinds of mental vocabulary to my car, and sometimes not very sympathetic when the thing won’t start. But we don’t take it seriously in supposing that the car has mental states in the same way that we do. I want to say the same thing about my computer. I ascribe all kinds of mental properties to it. It’s completely natural. The mistake is if we suppose that those are to be taken at face value.
But a third and really important reason, and I think the deepest reason, that leads to this mistake is oddly enough that in all of these discussions with all of their technical vocabulary, there is an old-style philosophical mistake that goes back to the seventeenth century-it goes back to Descartes, and frankly it goes back to Plato. This is the mistake-it is the refusal to think that the mind is just a biological phenomenon like any other. That is, in all of these discussions you get the idea that intelligence is something very abstract, that it can’t be just a fact of biology like digestion, or the secretion of bile, or mitosis or meiosis.
I want to say that it is a kind of contempt for biology or a willingness to ignore the obvious facts of biology that leads to what is essentially the dualist view that mental states and minds are just programs.
M. L. MINSKY: I’d like to say a couple of words about the things Dr. Searle said. This consciousness thing is a very complicated business. And Searle is imagining a machine that looks up rules and executes them, and he’s saying, “Well how would it feel?” But I think he’s confusing what’s going on in the process with what somebody else might feel when he’s looking at it. Of course, if you look at somebody’s brain cells with the right instrument, you’ll see the nervous impulses going around, and you can say it’s just adding and subtracting or whatever it’s doing with those electric currents. To think that you know what it feels like to be a typewriter and that if you were a typewriter, you wouldn’t feel anything, and to assume from that that if you were a machine a billion times as complex as a typewriter, you wouldn’t feel anything is an extraordinary extrapolation. It seems to me this is very similar to his remarks at the end about consciousness and digestion being biological. Everybody knows that there’s no vital mystery to digestion if you just know how the enzymes work. There isn’t, in a certain reductionist sense, any such thing as digestion.
Now you might say that I’m missing the point because that would mean there isn’t any such thing as consciousness. But my complaint is that the people who think there is such a thing as consciousness are trying to simplify it so much. Suppose I ask myself if I was aware four seconds ago of saying the word consciousness. What is consciousness to me? What does it mean to be self-aware? Now think of that Chinese machine that interpreted and answered questions in Chinese without knowing it. Suppose I asked it the question in Chinese, “Were you aware that four seconds ago you translated the word kumquat into mandarin orange or some erroneous such thing?” The machine, if it said yes-and Searle has to grant it says yes because he’s assuming that it did all of the things it needed to do to translate correctly-it certainly would have said, “Yes, I was conscious.” What does it mean to say that? If I say that I was conscious of doing something a few minutes ago, that means that somewhere in my data processing, I must have made a pale copy of the state of the machine. So you see it’s true that if the machine had no trace of its past state, like the old typewriters, then it can’t answer questions about how it felt and it wouldn’t mean anything to ask it such a question because you know that it’s amnesic-every second is disconnected from the next, there’s no trace. But if I translate a word and if I can say later, “Yes, I knew I did that,” I could only answer that if I made little fuzzy copies. What I’m saying is that the mind is very complicated. When I say I and when Searle says first person, there’s no such thing as first person. There are a lot of processes going on. If I ask, how did I feel when I met so and so and an hour ago, I go back and I run these processes, but it’s not that there was a way I felt. If you’ve read anything about the mind, you know that there are many parts of my mind, some of which I don’t know about for years, some of which I know right away. And what do I mean by I? You see, I’m falling into the trap. There are five parts of the mind. This one has some copy of the state of that one a while ago, this of that, this of that. When you say first person, when you say I felt, a little piece of machinery inside of all this that’s connected to the mouth and won’t stop, is pretending that it knows what happened there. That it was conscious. That it could be responsible for the things that went on in all the other parts. Well, this is all so absurd. A thing like saying a machine can’t feel pain really gets you into very complicated issues. It’s no use to say a syntactic process can’t feel because syntax is in fact the technical word for describing what doesn’t happen in a process.
Seymour Papert: One might have said that watches are definitionally made up of wheels turning, and something that doesn’t have wheels turning isn’t a watch. If one had a commitment to this point of view, obviously these digital things that many people are wearing would not be watches. They’d be simulations of watches. And they wouldn’t keep time. They’d do something like seeming to keep time.
Obviously, machines think not in the same way exactly as people think. And obviously people are biological. And when we say Is it true that machines think? we’re asking whether we would like to extend the notion of thinking to include what machines might do. That’s the only meaningful sense of the question, Do machines think? Newton said the sun exerts a force on the earth. And one might have said, “No, forces are what you do with your muscles, it’s biological,” but we’d be missing the point. Newton was introducing a new technical concept of force. And AI, if it’s going to be taken seriously, is introducing a new technical concept of thinking, one that is not the same as the concept of thinking that’s existed since Aristotle and before.
So I’ve got to agree, of course, that surely machines don’t think in exactly the same way as I do. I’m not sure Searle thinks in exactly the same way I do either, or whether any two people think in exactly the same way. But what is obvious to me is that machines don’t think in exactly the same way as me or in exactly the same way as one another. And one would deliberately-in working with artificial intelligence-try to make machines think in very different ways so as to have a comparative study of the different ways that this sort of function that we’re all interested in can operate.
So I think that Searle’s conclusion is true, but I don’t know what it’s relevant to. Of course, it might be that if you poke around in the artificial intelligence literature as he says, there really are some people who have said machines think in exactly the same way as people do. Maybe they meant it, maybe they didn’t mean it. Maybe this was a slip of the tongue. Maybe they were philosophically naive. All this is rather irrelevant I think to any fundamental considerations about the nature of thinking and whether something that is shared by us and potential machines is a more coherent, more useful, more powerful notion than this pretechnical sense of thinking that’s rooted in the biological.
H. R. PAGELS: Dr. Papert, I’m a little confused. Your remarks make a strong distinction between the kind of “thinking” that a computer might do and the kind of thinking that a human being does. You seem to suggest that these two kinds of thinking have nothing to do with one another. In other words, AI-type thinking has nothing to do with human thinking. But that was not the impression that I got from your earlier remarks.
Seymour Papert: Well, let’s take the example of Newton expanding the concept of force from someone pushing to something that the sun might do to the earth. He has extended the notion of force; he has changed it. To go from that to saying that his notion of force has nothing to do with what might happen when I push the table is absurd. I think that this is what the theory of intelligence is about, that we are constantly extending the theory or thinking, whether it’s in the psychology lab or in the philosophy seminar or in making machines. We are constantly extending and defining our notion of thinking and making new notions of thinking-technical ones. It’s not true to say they have nothing to do with the previous ones. So I don’t think that there’s anything in Searle’s argument that could be construed as trying to prove that what happens in the machine has nothing to do with what happens in the person. What he has argued for-and I’m accepting this for the sake of argument-is that it’s not exactly the same thing. And I think his argument depends essentially on the attempt to want it to be exactly the same thing.
H. R. PAGELS: So, if I understand you, there is a more general idea of thinking which includes both machine thinking and human thinking.
Seymour Papert: And the theoretical enterprise is trying constantly to extend.
J. MCCARTHY: I’d like to go back to the Chinese room. There is a confusion between the system consisting of the person and the person himself. I agree with Robert Wilensky who made the same point earlier. The system knows Chinese, but the person who is executing the system may not. This is analogous to an interpreter running in a computer, the interpreted program often has capabilities the interpreter does not. It’s just that we don’t have experience with systems in which a person carries out a mental process that has properties different from those of the person himself. We get the same confusion with computers. Someone asks me whether LISP can do calculus problems. No, LISP cannot do calculus, but some LISP programs can.
The example of thirst is different. A program that simulates thirst is not going to be thirsty. For example, there is no way to relieve it with real water.
Searle has said that the people in AI take the third-person view of mental qualities. That’s correct. We do, and we’ll claim that it’s a virtue. He says we consider the problem of intelligence as distinct from biology. Yes, we hold that intelligence is something that can be dealt with abstractly just as computation can be discussed and dealt with abstractly. One can ask whether a computer calculates the sum of 3 and 5 to be 8 in the same sense as a human does. I suppose Searle would agree that “calculate” is being used in the same sense for both human and machine in this case.
Now there’s the point Dreyfus made about it taking 300 years. I have been saying that human-level AI will take between 5 and 500 years. The problem isn’t that it will take a long time to enter data into a computer. It is rather that conceptual advances are required before we can implement human-level artificial intelligence-just as conceptual advances were required beyond the situation in 1900 before we could have nuclear energy.
Pursuing the nuclear energy analogy, the question is whether the present AI situation corresponds to 1900 or to 1938 when Rutherford, the leading nuclear physicist, declared nuclear energy impossible. The situation of 1938 is interesting in that experiments exhibiting nuclear fission had already been done but had been misinterpreted. Perhaps someone has already done experimental research that, when properly interpreted, will make possible human-level AI. I would be very surprised. When we talk about future conceptual advances, we don’t know where we stand at present.
Dreyfus made a point about a reservation machine not knowing whether 6:25 will do as a little earlier than 6:30. The program would have the same problem if it were making a reservation for a robot. Whether even a 6:29 reservation will do depends on circumstances. So the fact that the reservation is for humans isn’t the problem.
Finally, let me defend Searle on one point. He was discussing whether a computer can think in the same sense as a human-not does it think in the same way. In my opinion the thermostat thinks the room is too warm in the same sense as a human might, and he would disagree. Likewise about whether the dog simulation wants to go out.
H. L. DREYFUS: I want to be very brief because I think there are two separate issues and the bulk of the questions have been directed at John Searle’s issue. Let’s distinguish the issues. That will help people be clear what’s going on. There’s the question of whether programmed computers of the sort that we now have with the sorts of programs that we now have could ever behave like human beings. And I want to say they can’t. And then there’s the question, Even if computers behaved exactly like human beings, would they be thinking? Would they be intelligent? Would they have meaningful mental states? Searle wants to grant the first point hypothetically, i.e., that programmed as at present, computers could be intelligent, and then to say even if they behave exactly like human beings, that still wouldn’t tell us anything about thinking because such machines wouldn’t be thinking.
To return to issue one, McCarthy brought up the conceptual advances required before we reach the level of commonsense artificial intelligence. I agree we need some breakthroughs, but it all hinges on what you mean by conceptual advances. I think we’re not in the same position as we were with respect to atomic energy in 1900 or in 1938, but more like the alchemists were with respect to atomic energy. That is, it’s not just a question of the right conceptual advances. We’re not even in the right dimension. We’re trying to use computers that have programs. And the programs operate using facts and rules stored in complicated data structures. I just think that that’s not going to get us common sense because that’s not the kind of knowledge that gives us common sense. It seems to me highly unlikely that we could ever cash what gives us common sense into enough knowledge to make a computer seem to have common sense.
The travel agent might have the same sort of problem I mentioned with Martians as with robots. Of course, if Martians were enough like us, the travel agent would not have the same problem, but if they lived 100 times faster, or died in a day like mayflies, then their concerns about a little earlier and a little later would be very different from ours. And so you have to understand how human beings live in time or how any other kind of creature lives in time to see what kind of problems it has and the best way to know how it experiences time is to be of the same species.
J. SEARLE: Can a machine think? Well, I want to tell you that’s a fairly tricky question. So let’s slow down and go through it. There is a sense in which each of us is a machine. We’re each a material system. We just have an awful lot of neurons up there. And in that sense, it seems to me the answer to the question is obvious-of course machines can think; we are thinking machines.
So maybe we’re trying to ask another question. We’re trying to ask the question, could an artifact think? Could you make a thinking machine? But there again, I don’t see any difficulty in principle. Suppose we got the billions of neurons with their axons and their dendrites and synaptic clefts, and neurotransmitters and all the rest of it. If you can duplicate the causes, then you can duplicate the effects. So that wasn’t the question we’re trying to ask.
Here’s another version of the question-Could a digital computer think? We’re getting closer now. Even that’s a little bit tricky because we want to say just about any system has a level of description where you can describe it as a digital computer. You can describe it as instantiating a formal program. So in that sense, I suppose, all of our brains are digital computers, and in that sense a digital computer can think.
The question we’re driving at is this-and that is really the heart of the matter-Could a system think solely by virtue of being a digital computer? That is to say, solely by virtue of instantiating the right program with the right inputs and the right outputs? And there the answer is no. And it has to be no for the reasons that I said earlier. Namely, the purely formal processes can’t by themselves give you the content. The same formal processes can be instantiated in any number of different kinds of substances which have quite different biochemical features, most of which will simply be incapable of duplicating the powers of the brain.
There are two very simple axioms on which my whole argument rests. One, the brain causes mental states. And just as a slogan, “brains cause minds.” If we know anything about the world, we know that much about how it works.
Two, formal processes by themselves are not sufficient for semantic or mental content. I put that in the slogan by saying “From syntax, you can’t get semantics.” I don’t mean to confine it to linguistics. That’s just a mnemonic for reminding us of the difference between the formal, the purely formal, and the content.
So the question we were trying to ask was, could a system think solely by virtue of being a digital computer? And the answer to that question has to be no. But now we can make a derivation from these two axioms-number one, brains cause minds, and number two, from syntax alone, you don’t get semantics. What follows is that the way the brain does it can’t be by instantiating a computational program alone. It can’t. When we explain how the brain produces mental states, we will not be able to do it entirely by the fact that the brain instantiates formal programs. And indeed, where we actually know something about the operations of the brain, we don’t have to appeal to a formal or abstract level. What we do appeal to is quite specific biochemical facts about the operation of the brain as we appeal to quite specific biochemical facts about the operation of the digestion or the operation of the liver or the pancreas or anything else.
Now there are two other little points I want to take up. Throughout these discussions, we tend to think there must be some technical solution to this problem. If you watch the discussion, people will often say that we get so far with computers, but there’s always that extra little bit we can’t go. So, they will say, “Can you program a computer that will fall in love, have a sense of humor, or whatever?” But I want to say that that really misses the point. The point is not that the digital computer doesn’t get quite all the way to having a mind, it doesn’t get started. It’s not in that line of business.
You can always say, of course, “Well it’s just an extended notion of thinking. Why not have a larger notion of thinking?” I have no objection to using words in an extended sense provided you make clear the extension. Now I want to say computers think and have thoughts and feelings and consciousness in the same extended sense of feelings, consciousness, and thought that computer simulations of rainstorms leave us all drenched, or computer simulations of five-alarm fires burn all the buildings down. It’s only an extended sense of leaving us all wet or burning the building down. I have no objections to that kind of talk, provided you realize that there is a sense in which it just abandons the claim of strong AI.
Indeed, I want to conclude by saying, why does anybody feel tempted to adopt strong AI? That is, McCarthy has written, and I quote this verbatim because it made a big impression on me, “Even a machine as simple as a thermostat can be said to have beliefs.” And he means that quite literally-beliefs in the same sense that you and I have. I’ve discussed this with McCarthy enough to know that he thinks the thermostat literally has beliefs. I once asked him what beliefs the thermostat had? And he told me the thermostat has three beliefs. It believes it’s too cold in here, it’s too hot in here, and it’s just right in here.
What I want to conclude with is this-it seems to me unnecessary, in order to pursue what I think is the really fruitful part of artificial intelligence, for people to adopt strong AI. I mean the computer is a wonderfully exciting tool. And computer science is a very exciting field of human investigation. It isn’t necessary either for the success of artificial intelligence or even for the getting of substantial research grants that we should make exaggerated claims.
H. R. PAGELS: I want to move the discussion in a different direction. I’d like to know the panelists’ viewpoints on where this discussion might be in 5 or 10 years. Is it possible that an advance in neurobiology, computer design, or the conceptual foundations of artificial intelligence research might resolve this issue one way or the other?
Another area we might discuss is potential collaboration between philosophers and people working in artificial intelligence research.
M. L. MINSKY: I don’t have much to say about that. I think as things are discovered and demonstrated, then-barring paradigm shifts of very large magnitude-attitudes will drift a little bit when machines seem more intelligent, people will tend to think that they’re more intelligent or that they’ll think.
But there are things that never change until there’s a paradigm change. For example, very often someone will ask me, “What is intelligence?” And I’ll say, “Well I don’t know I’m trying to find out.” And they’ll say: “Well I didn’t mean that. I mean how do you define it? How do you define intelligence?” And I go back and say, “Well how do you define life?” The word “life” doesn’t have much place anymore in science, as many of you know. There’s a sort of continuum. We know that crystals can copy certain patterns and that a million billion years ago, events started to happen where these very complicated systems started appearing. And if you ask a biologist what he’s studying, he’ll say he’s studying some facet of the digestion microworld. To develop a theory to understand a complicated system, we have to cover very small areas until it’s all covered up and then sew it all together with exceptions or other theories. I think what will happen in artificial intelligence is that to a large extent, these side issues will decay. Obviously the machines that we have now, whether they think or not in the weakest sense that John McCarthy suggested, are nothing at all like us. They can’t remember why they did things. People put in programs to help them. And so, it makes very little sense to talk about them having much in the way of consciousness or sensitivity or whatever.
As for 10 years or 200 from now, we don’t know. I want to point out that we don’t have to spend all the time of the future figuring out what common sense is. We may be able to make learning machines that will cleverly watch their own behavior and decide that certain principles they’re using don’t work and edit them.
Gradually, the attitude and the respect for intelligent machines will change as they become more intelligent. But in our culture, until there’s a revolution, the idea will persist that they’re just simulations. People will say, “It’s crying terribly out there, that robot. Don’t you think you should let it in?” And other people will say, “No it’s just simulating. It doesn’t really feel bad.” I wonder if Searle thinks that the brain gets wet when it thinks about rain. Whatever the process is, why should it matter what the substance is as long as the impulses are there. When we look at a picture of a pretty girl, let’s face the fact that we’re not seeing a pretty girl, we’re only seeing a lot of little colored dots on the retina. And then let’s face the next fact, when we see a real pretty girl, we’re still just seeing little dots on the retina, if you confine the system to that. I think that’s what one means by you can’t just look at the interpretation process, you have to look at the system of the person carrying out the rules. We never see anything. Does that mean that we do never see anything? It just depends on the size of the system to which you make this attribution.
So, unless there’s a revolution that says words like “life” should be removed, everybody will still feel there’s a difference. That these robots are maybe intelligent but not alive, or that they have pseudointelligence but not real. And the attributions that you use there will depend on your purposes. Are you worrying about them getting legal rights and inheriting your property? Are you worried about having to share? Are you worried about feeling guilty that they might have the same feelings as you although maybe John Searle is right and one could never prove such a thing? Because it’s just a style of thought itself. These questions will depend on too much to answer.
Seymour Papert: I’d like to say something about the question of where we are going, and maybe this is relevant to what kind of cooperation there might be between AI and the different kind of philosopher-a philosopher concerned with reality, with social reality and with people’s concerns.
I’d like to put this into a different context. Can machines be intelligent? Can they think? There’s a certain form of this question that is of very vital concern to all of us, because there will be machines that will be better than people at doing whatever their jobs are. If your job is being a doctor, there will be a machine that can make a better diagnosis and decide on a better treatment of your patient than you can. Or if you want to invest in the stock exchange, rather than go and ask a human advisor, you should get a machine. The machine will know much better than you-probably will have made millions of dollars already. I think that most people are more concerned about this than they are willing to admit. Because it really is frightening.
Now given that people think this is even slightly likely, they might want to be reassured. And I believe that a lot of people have found reassurance in the writings of Dreyfus and Searle. They read what these philosophers say as somehow reassuring that this isn’t going to happen, that people are always going to keep their unique special position in the universe, that maybe the diggers of ditches and the drivers of horses have had their jobs taken away from them, but we intellectuals will always have our jobs and prestige and all the things that come with our particular kind of work.
Can machines ever make us intellectuals obsolete? Dreyfus has said no, they can’t because there are certain things that machines will never be. And Searle takes a slightly different position, saying that whatever machines do, we won’t call it thinking but we can’t define what they’ll do or won’t do from the outside.
I think it’s a little bit chancy to make predictions about what machines can’t do, because some clever engineer will come along and make the machine do that thing. So, I think a lot of people would rather have something like the Chinese room argument which doesn’t depend on competing with the technicians about what is technically possible. Undoubtedly this kind of argument has acquired a certain social popularity for that reason.
But I would like to say, concerning the consequences, that I feel that this is a very dangerous situation. I think it’s socially dangerous for people to be lulled into a sense of security that there won’t be machines to threaten their positions. I think that these are important issues of what it really means to our sense of ourselves to allow ourselves to face the social issue of what machines will be doing out there-what life will be like-after 300 years or 3 years or whatever.
To go back to a point, I made at the beginning—that machines are changing the way people think because machine thinking is often useful as a model for human thinking. It’s also sometimes dangerous as a model. For example, if you think that Newell and Simon machine that solves puzzles in a certain way is a good model for us to follow, you might be concerned about whether this is a good model for our children to follow. Finding the appropriate context for discussing that is a matter of vital social concern, and it’s not dealt with-it’s only covered over-by saying, “Well that machine is not thinking because it’s not biological, or because it’s just following rules, or because it’s not semantic.” The point is, you give it a problem, it gives you an answer. You ask it why it did this and it gives you reasons. And then you’re going to tend to follow that kind of reason or not follow that kind of reason. Or pass them on to your children or not pass them on to your children. I’m just suggesting this as somewhere that AI is going. And the kinds of consideration that have come up here can be dangerously misleading if we don’t keep them rigorously separated from all issues of this kind.
J. MCCARTHY: The question is, what will be the situation 5 to 10 years from now? Let me make it 10 or 15. I think there’ll be a paradigm shift among the public that will give John Searle the following problem. He will want to come to the symposium to correct our use of mental terms, but he won’t even get here, because he’ll have to correct his secretary who will tell him, “It promised to process your travel advance, but I don’t think it will, because it’s puzzled about whether the expenditure for flowers was intended and necessary for the business’s goals.”
Thus in 10 or 15 years, quite mundane systems used for business and personal purposes will require the use of a certain amount of mental terminology in order to be used effectively.
Also let me repeat my warning to philosophers that if they insist on discussing commonsense reasoning only at the general level of today’s discussion, they will lose the jurisdiction. We need to consider the conditions for the ascription of particular mental qualities, and this may require collaboration among philosophers and artificial intelligence researchers.
We attempted such a collaboration several years ago, but I think the particular attempt was unsuccessful largely because it considered overly general questions. This was partly because the AI people succumbed to the temptation to become amateur philosophers rather than raising the AI issues to which philosophy is relevant.
H. L. DREYFUS: I think that I’m the one, not John Searle, who’s lulling people into a sense of security. I just think it’s not a false sense of security. I think that you should be confident and trust your natural intuitions and your grandmother’s natural intuitions that you are not a machine. Not that you are not a material thing, you certainly are. But you are not the sort of machine that, by using a program manipulating facts by way of rules, produces the kind of behavior we call intelligent.
Now I want to make clear that that’s not to say-as people have rightly said that computers won’t fall in love, etc. A recent Time Magazine essay, claiming to be giving my views, said that I think that computers won’t be able to pray, won’t be able to look you in the eye while shaking hands, and so forth. Of course I think that. But that’s not important. What I want to say is that even though computers can do really complicated things like analyze spectrograms and play chess, computers as we now understand them and program them can’t even understand the sort of stories that four-year-old children understand. Moreover, I think they will never be able to understand the sort of stories that four-year-old children understand because such stories involve emotions like jealousy, everyday practices, seeing one situation as similar to another situation, and the sort of rules that include a clause saying, “everything else being equal,” without spelling out what everything else is, or what counts as being equal. It’s that sort of thing, it seems to me, that the current way of trying to produce artificial intelligence will never achieve. And so I don’t think computers will ever behave like human beings if the current direction of research continues. And you can be secure that there won’t be the sort of robots around doing the sort of things we do and thinking like we do in your lifetime or even in the foreseeable future.
Let me take the rest of my time to try to answer Dr. Pagel’s question, where will we be in 10 years? When I wrote What Computers Can’t Do 10 years ago, nobody seemed to think that the problem of making computers intelligent required programming commonsense knowledge. People were trying to use shortcuts to get computers that seemed to be intelligent. Now, 10 years later, everyone thinks there is a big problem concerning commonsense knowledge. Where will we be 10 years from now? I think 10 years from now people might realize that it’s not a question of commonsense knowledge at all. That common sense includes feelings, having a body, having images, responding to similarities, etc. Then maybe the field will switch to something that I feel would be more promising, simulating the brain by neural nets with changing thresholds rather than storing facts and rules. And in that new dimension, in x number of years, maybe we’d get somewhere. But I don’t think we’ll get somewhere by trying to treat our commonsense understanding as if it were knowledge.
J. SEARLE: I always have the disconcerting sense in these discussions that I’m busy saying over and over, “Look, 2 plus 2 is really just 4,” while other people are saying, “Yes, but if 2 plus 2 were 5 or 7, think of the terrific derivations that we could make.”
Now what will this discussion look like in 10 or in x number of years? I have very great confidence in human rationality. People are going to say, “Yes, 2 plus 2 is just plain 4.” And the specific form I think that’s going to take is this. Mentality is a biological phenomenon. We don’t know an awful lot about how it works in the brain but suppose we really did know how it worked. Suppose we had a perfect science of the brain or even a pretty good science of the brain. Suppose we were able to explain how the behavior at the neuronal level caused consciousness and thought processes at the same degree that we can now explain how the behavior of H2O molecules causes the liquidity of the water in this glass. Once we get to that point, then nobody will make these confusions. Ignorance is one of the reasons that lead to strong AI. Notice what the strong AI partisans are saying. They’re saying the brain doesn’t matter. And that is literally incredible. It is incredible to suppose that the brain doesn’t matter to the mind, that any system whatever, whether it’s beer cans or, as Weizenbaum said, rolls of toilet paper with stones laid on the squares, any system at all will have to have mental states in exactly the same sense that you and I do because all there is to having mental states is instantiating the right program.
I believe this discussion will peter out when people realize that that’s a preposterous view.
Another reason that it’s still possible to propound strong AI is that there’s still a certain mystique surrounding the computer. We’re not as at home with the computer as we are with cars and telephones and so on. It still seems for most of us a kind of mysterious object, and that leads to a kind of mystification. And that will peter out as the computer becomes more common.
So, my answer to the question of what is likely to happen over the years is that people will see that strong AI, the idea that the computer program is sufficient for having mental states, is in a way a last gasp of a Cartesian metaphysics. The last gasp for the idea that the mind is something special. That it isn’t part of the biological universe like the rest of the biological facts about our life.
I’d like to end on a more constructive note. I do see enormous possibilities for collaboration between AI and the sort of philosopher that I am. In fact, I’ve engaged in some of this. And this seems to me the real world of AI. Suppose you have a language and certain rules for speech acts and you also have rules for understanding indirect speech acts, that is, rules for understanding how it is that when a guy says, “You’re standing on my toe,” you know perfectly well that’s not just a remark about your geographical location. He’s actually trying to get you to do something about it. I think those are rational processes. Now the beauty of AI, and this I really do admire, is that it forces you to pose those questions precisely and forces you to state your theory precisely. In fact, the things I’ve written about-metaphors and indirect speech acts and so on-a great deal of it has been programmed by people working in various AI labs. So I think, in fact, that AI is an immensely useful tool in the study of language and the study of the mind, just as it’s a useful tool in the study of rainstorms or the economy or anything else. I feel fairly confident that in the end, the exaggerated and I think implausible and preposterous kinds of strong AI will be abandoned.
What McCarthy says is surely right, we will come to make all these attributions of intentional states to computers as we now make attributions of mental states to cars and adding machines and photoelectric cells and so on. It’s quite harmless. Nobody supposes that his car literally has thoughts and feelings. And in the same way, we will come to the point where we realize that our PDP-10 or even our beloved Apple II doesn’t actually have thoughts and feelings. And when that happens, then we’ll be able to see what is really useful in AI, namely, it is itself a tremendous intellectual advance in the development of computer science and, even more important, it enables us to make further intellectual advances in other fields.