Source: Minsky, M., & Papert, S. (1972). Artificial Intelligence Progress Report (MIT Artificial Intelligence Laboratory Artificial Intelligence Memo, Issue. 

MIT ARTIFICIAL INTELLIGENCE LABORATORY

Artificial Intelligence Memo No. 252  |  January 1, 1972

Artificial Intelligence Progress Report

by Marvin Minsky and Seymour Papert

DOCUMENT METADATA & ACKNOWLEDGMENTS Research at the Laboratory in vision, language, and other problems of intelligence. This report is an attempt to combine a technical progress report with an exposition of our point of view about certain problems in the Theory of Intelligence. It is identical with pp. 129–224 of the 1971 Project MAC Progress Report VIII. This work was conducted at the Artificial Intelligence Laboratory, a Massachusetts Institute of Technology research program supported in part by the Advanced Research Projects Agency of the Department of Defense and monitored by the Office of Naval Research under Contract number N00014-70-A-0362-0002 and in part by the National Science Foundation under grant number GJ-1049.

The M.I.T. Artificial Intelligence Laboratory

The A. I. Laboratory is concerned with understanding the principles of Intelligence. Its goal is to develop a systematic approach to the areas that could be called Artificial Intelligence, Natural Intelligence, and Theory of Computation. Here are its main current foci of attention:

• ARTIFICIAL INTELLIGENCE: Robotics; Vision, mechanical manipulation, advanced automation. Models for learning, induction, analogy. Schemata for organizing bodies of knowledge. Development of “heterarchical” program control structures.
• NATURAL INTELLIGENCE: Models of structures involved in “commonsense thinking”. Understanding meanings, especially in natural language narrative. A new educational methodology, based on development of the child’s abilities to describe processes.
• THEORY: Computational trade-offs between time, memory size, and processor parallelism. Study of computational geometry as a tool for comparing different structures and strategies. Theory of schemata, for analysis of complexities of certain algorithms and languages.

These subjects are all closely related. The natural language project is intertwined with the commonsense meaning and reasoning study, in turn essential to the other areas, including machine vision. Our main experimental subject worlds, the “blocks world” robotics environment and the children’s story environment, are better suited to these studies than are the puzzle, game, and theorem-proving environments that became traditional in the early years of artificial intelligence research. Our evolution of theories of intelligence has become closely bound to the study of development of intelligence in children. The educational methodology project is symbiotic with the other studies, both in refining older theories and in stimulating new ones; we hope this project will develop into a center like that of Piaget in Geneva.

As it has crystallized over the past few years, the main elements of our viewpoint can be summarized cryptically:

CORE VIEWPOINT Thinking is based on the use of SYMBOLIC DESCRIPTIONS and description-manipulating processes to represent a variety of kinds of KNOWLEDGE — about facts, about processes, about problem-solving, and about computation itself, in ways that are subject to HETERARCHICAL CONTROL STRUCTURES in which control of the problem-solving programs is affected by heuristics that depend on the meanings of events.

The ability to solve new problems ultimately requires the intelligent agent to conceive, debug, and execute new procedures. Such an agent must know to a greater or lesser extent how to plan, produce, test, modify, and adapt procedures; in short it must know a lot about computational processes. We are not saying that an intelligent machine, or person, must have such knowledge available at the level of overt statements or consciousness, but we maintain that the equivalent of such knowledge must be represented in an effective way somewhere in the system.

This report illustrates how these ideas can be embodied into effective approaches to many problems, into shaping new tools for research, and into new theories we believe important for Computer Science in general as well as for Robotics, Semantics, and Education.

1.0 Vision and Description

When we enter a room, we feel we see the entire scene. Actually, at each moment most of it is out of focus, and doubly imaged; our peripheral vision is weak in detail and color; one sees nothing in his blind spot; and there are many things in the scene we have not understood. It takes a long time to find all the hidden animals in a child’s puzzle picture, yet one feels from the first moment that he sees everything. People can tell us very little about how the visual system works, or what is really “seen”. One explanation might be that visual processes are so fast, automatic, and efficient that there is no place for introspective methods to operate effectively. We think the problem is deeper. In general, and not just in regard to vision, people are not good at describing mental processes; even when their descriptions seem eloquent, they rarely agree either with one another or with objective performances. The ability to analyse one’s own mental processes, evidently, does not arise spontaneously or reliably; instead, suitable concepts for this must be developed or learned, through processes similar to development of scientific theories.

Most of this report presents ideas about the use of descriptions in mental processes. These ideas suggest new ways to think about thinking in general, and about imagery and vision in particular. Furthermore, these ideas pass a fundamental test that rejects many traditional notions in psychology and philosophy: If a theory of Vision is to be taken seriously, one should be able to use it to make a Seeing Machine!

1.1 Reasoning by Analogy

To emphasize that we really mean “seeing” in the normal human sense, we shall begin by showing how a computer program — or a person — might go about solving a problem of “reasoning by analogy”. This might seem far removed from questions about ordinary “sensory perception”. But as our thesis develops, it will become clear that there is little merit in trying to distinguish “sensation” or “perception” as separate and different from other aspects of thought and knowledge.

When we give an educated person this kind of problem from an IQ test, he usually chooses the answer “figure 3”.

People do not usually consider such puzzles to be problems about “vision.” But neither do they regard them as simply matters of “logic”. They feel that other, very different mental activities must be involved. Many people find it hard to imagine how a computer program could solve this sort of problem. Such reservations stem from feelings we all share: that choosing an answer to such a question must come from an intuitive comprehension of shapes and geometric relations, rather than from the mechanical use of some rigid, formal rules.

However, there is a way to convert the analogy problem to a much less mysterious kind of problem. To find the secret, one has merely to ask any child to justify his choice of Figure 3. The answer will usually be something like this:

CHILD’S EXPLANATION “You go from A to B by moving the big circle down. You go from C to 3 in the same way by moving the big triangle.”

On the surface this says little more than that something common was found in some transformations relating A with B AND C with 3. As a basis for a theory of the child’s behavior it has at least three deficiencies:

• It does not say how the common structure was discovered.
• It appears to beg the question by relying on the listener to understand that the two sentences describe rules that are identical in essence although they differ in details.
• It passes in silence over the possibility of many other such statements (some choosing different proposed answers). For example, the child might just as well have said:
  “You go from A to B by putting the circle around the square…” or
  “You go from A to B by moving the big figure down,” etc.

Aha! If that last statement were applied also to C and 3 the rules would in fact be identical! This leads us to suggest a procedure for a computer and also a “mini-theory” for the child:

Step 1. Make up a description DA for Figure A and a description DC for C.
Step 2. Change DA so that it now describes FIGURE B.
Step 3. Make up a description D for the way that DA was changed in step 2.
Step 4. Use D TO CHANGE DC. If the resulting description describes one of the answer choices much better than any of the others, we have our answer. Otherwise start over, but next time use different descriptions for DA, DC and (perhaps) for D.

Notice that Step 3 asks for a description at a higher level! The descriptions in Steps 1 and 2 describe pictures, e.g., “There is a square below a circle”. The description in Step 3 describes changes in descriptions, e.g., “the thing around the upper figure in DA is around the lower figure in DB.” Our thesis is that one needs both of these kinds of description-handling mechanisms to solve even simple problems of vision. And once we have such mechanisms, we can easily solve not only harder visual problems but we can adapt them to use in other kinds of intellectual problems as well — for learning, for language, and even for kinesthetic coordination.

This schematic plan was the main idea behind a computer program written in 1964 by T. G. Evans. Its performance on “standard” geometric analogy tests was comparable to that of fifteen-year-old children! This came as a great surprise to many people, who had assumed that any such “mini-theory” would be so extreme an oversimplification that no such scheme could approach the complexity of human performance. But experiment does not bear out this impression. To be sure, Evans’ program could handle only a certain kind of problem, and it does not become better at it with experience. Certainly, we cannot propose it as a complete model of “general intelligence.” Nonetheless, analogical thinking is a vital component of thinking, hence having this theory [Evans, 1964], or some equivalent, is a necessary and important step.

In developing our simple schematic outline into a concrete and complete computer program, one has to fill in a great deal of detail: one must decide on ways to describe the pictures, ways to change descriptions, and ways to describe those changes. One also has to define a policy for deciding when one description “fits much better” than another. One might fear that the possible variety of plausible descriptions is simply too huge to deal with; how can we decide which primitive terms and relations should be used? This is not really a serious problem. Try, yourself, to make a great many descriptions of the relation between A and B that might be plausible (given the limited resources of a child) and you will see that it is hard to get beyond simple combinations of a few phrases like “inside of”, “left of,” “bigger than,” “mirror-image of,” and so on.

But let us postpone details of how this might be done [see Evans, 1964] and continue to develop our central thesis: by operating on descriptions (instead of on the things themselves), we can bring many problems that seem at first impossibly non-mechanical into the domain of ordinary computational processes.

WHAT IS A DESCRIPTION? What do we mean by “description”? We do not mean to suggest that our descriptions must be made of strings of ordinary-language words (although they might be). The simplest kind of description is a structure in which some features of a situation are represented by single (“primitive”) symbols, and relations between those features are represented by other symbols or by other features of the way the description is put together. Thus the description is itself a MODEL — not merely a name — in which some features and relations of an object or situation are represented explicitly, some implicitly, and some not at all. Detailed examples are presented in 4.3 for pictures, and in 5.5 for verbal descriptions of physical situations. In 5.6 there are some descriptions which resemble computer programs. If we were to elaborate our thesis in full detail we would put much more emphasis on procedural (program-like) descriptions because we believe that these are the most useful and versatile in mental processes.

1.2 Children’s Use of Descriptions

The theory of analogy we have just proposed might seem both too simpleminded and too abstract to be plausible as a theory of how humans make analogies. But there is other evidence for the idea that mental visual images are descriptive rather than iconic. Paradoxically, it seems that even young children (who might be expected to be less abstract or formal than adults) use highly schematic descriptions to represent geometric information.

We asked a little boy of 5 years to draw a cube. “Very good,” we said, and asked: “How many sides has a cube?” “Four, of course,” he said.

“Of course,” we agreed, recognizing that he had understood the ordinary meaning of “side,” as of a box, rather than the mathematical sense in which top and bottom have no special status. “How many boards to make a whole cube, then?” “Six,” he said, after some thought. We asked how many he had drawn. “Five.” “Why?” “Oh, you can’t see the other one!”

Then we drew our own conventional “isometric” representation of a cube. We asked his opinion of it. “It’s no good.” “Why not?” “Cubes aren’t slanted!”

Let us try to appreciate his side of the argument by considering the relative merits of his “construction-paper” cube against the perspective drawing that adults usually prefer. We conjecture that, in his mind, the central square face of the child’s drawing, and the four vertexes around it, are supposed in some sense to be “typical” of all the faces of the cube. Let us list some of the properties of a real three-dimensional cube:

• Each face is a square.
• Each face meets four others.
• All plane angles are right angles.
• Each vertex meets 3 faces.
• Opposite edges on faces are parallel.
• All trihedral angles are right angles, etc.

Now, how well are these properties realized in the child’s picture?

• Each face is a square.
• The “typical” face meets four others!
• All angles are right!
• Each typical vertex meets 3 faces.
• Opposite face edges are parallel!
• There are 3 right angles at each vertex!

But in the grown-up’s pseudo-perspective picture we find that:

• Only the “typical” face is square.
• Each face meets only two others.
• Most angles are not right.
• One trihedral angle is represented correctly in its topology, but only one of its angles is right.
• Opposite edges are parallel but only in “isometric,” not in true perspective.

And so on. In the balance, one has to agree that the geometric properties of the cube are better depicted in the child’s drawing than in the adult’s! Or, perhaps, one should say that the properties depicted symbolically in the child’s drawing are more directly useful, without the intervention of a great deal more knowledge.

One could argue that in the adult’s drawing, the square face and the central vertex are understood to be “typical.” We gave him the benefit of the doubt. Also, one never sees more than 3 sides of a cube, but children don’t seem to know this, or feel that it is important. The parallelisms and the general “four-ness” surely dominate.

Incidentally, we do not mean to suggest that our child had in his mind anything like the graphical image of his drawing, but rather that he has a structural network of properties, features, and relations of aspects of the cube, and that what he drew matches this structure better than does the adult’s more iconic picture. In 4.4 we will show how such structural networks can be used in a program that learns new concepts as a result of experience.

Not all children will draw a cube just this way. They usually draw some arrangement of squares, however, and this sort of representation is typical of children’s drawings, which really are not “pictures” at all, but attempts to set down graphically what they feel are the important relations between things and their parts.

Thus “a ring of children holding hands around the pond” is drawn with people radiating outward from a central circle, perhaps because the correct perspective view would put some of the children in the water. Also, in the child’s drawing the people are all at right angles to the ground, as they should be! For the same reason, perhaps, “Trees on a mountain” is drawn with trees perpendicular to the sloped mountain face because trees usually grow straight out of the ground. It doesn’t matter if an actual scene is right in front of the child; he will still draw the trees sideways!

A person is often drawn with a large head and stick limbs coming directly off it, perhaps partly because the body that is so important to the adult doesn’t really do much for the child except get in his way, partly because it does not have an easily-described shape.

From all this we are led to a new view of what children’s drawings mean. The child is not trying to draw “the thing itself”; he is trying to make a drawing whose description is close to his description of that thing — or, perhaps, is constructed in accord with that description. Thus the drawing problem and the analogy problem are related.

We hope no reader will be offended by the schematic simplicity of our discussion of “typical children’s drawings”. Certainly we are focusing on some common phenomena, and neglecting the fantastic variety and plasticity of what children do and learn. Yet even in that plasticity we see the dominance of symbolic description over iconic imitation.

Most children before 5 or 6 years old draw people like this. Find such a child and ask him, “Where is his hair?” and draw some, or say “Why doesn’t his nose stick out?” and draw an angular line in the middle of the face. Chances are that if the child pays any attention at all and likes your idea, these features will appear in every face he draws for the next few months. The hair is obviously symbolic. The new nose is no better, optically, than the old, but the child is delighted to learn a symbolism to depict protrusion.

There is a vast literature describing phenomena and theories of “learning” in terms of the gradual modification of behavior (or behavioral “dispositions”) over long sequences of repetition and tedious “schedules” of reward, deprivation and punishment. There is only a minute amount of attention to the kind of “one-trial” experience in which you tell a child something, or in which he asks you what some word means. If you tell a child, just once, that the elephants in Brazil have two trunks, and meet him again a year later, he may tell you indignantly that they do not.

The success of Evans’ program for solving analogy problems does not prove anything, in a strict sense, about the mechanisms of human intelligence. But such programs certainly do provide the simplest (indeed, today the only) models of this kind of thinking that work well enough to justify serious study.

It is natural to ask whether human brains “really” use symbolic descriptions or, instead, manage somehow to work more “directly” with something closer to the original optical image. It would be hard to design any direct experiment to decide such a question in view of today’s limited understanding of how brains work. Nevertheless, the formalistic tendencies shown in the children’s drawings point clearly toward the symbolic side. The phenomena in the drawings suggest that they are based on a rather small variety of elementary object-symbols, positioned in accord with a few kinds of relations involving those symbols, perhaps taken only one or two at a time. These phenomena are not seen so clearly in the pictures of sophisticated artists, but even so we think the difference is only a matter of degree. While it is possible to train oneself to draw with quantitative accuracy some aspects of the “true” visual image, the very difficulty of learning this is itself an indicator that the symbolic mode is the more normal manner of performance. Even sophisticated adults often show a preference for unreal but tidy “isometric” drawings over more “realistic” perspective drawings, even though a cube is never seen exactly as in an isometric projection. In any case, all this suggests that “graphic” visual mechanisms become operative later (if at all) in human intellectual development than do methods based on structural descriptions. This conclusion seems surprising because in our culture we are prone to think of symbolic description as advanced, abstract, and intellectual, hence characteristic of more advanced stages of maturation.

2.1 Appearance and Illusion

Now consider some phenomena that might seem to be more visual, less intellectual. Diagonal stripes placed over a standard rectangle affect its appearance so that (to most people) the sides appear to lean out and no longer seem perfectly parallel. Such phenomena have been studied with great intensity by psychologists. Similarly, adjacent background shading can alter brightness perception so that two identical central grey squares appear to have entirely different shades.

A good deal is known about the effects of nearby figures or backgrounds on another figure. Perhaps most familiar is the Muller-Lyer phenomenon in which the directions of oblique arrow segments make horizontal lines of equal length appear unequal.

But the strangest illusion of all is this: to many psychologists these phenomena of small perceptual distortions have come to seem more important than the question of why we see the figures at all as “rectangle,” or “square,” or as “double-headed arrow!” Surely this problem of how we analyze scenes into familiar objects is a more central issue.

Thus one finds much more discussion about why a figure looks larger in a perspective context than about why one sees the figures as people at all. We agree that the study of distortions, ambiguities, and other “illusions” can give valuable clues about visual and other mechanisms. To resolve two or more competing theories of vision, such evidence might become particularly useful. First, however, we need to develop at least one satisfactory theory of how “normal” visual problems might be handled, particularly scenes that are complicated but not especially pathological.

Let us look at a few more visual phenomena. Consider drawings of pyramid-bases — simple flat-surfaced five-faced bodies that could be pyramids with their tops cut off. Two drawings may appear at first sight to be reasonable pictures of pyramid-bases, but detailed geometric analysis reveals that one of them cannot be a picture of such a body because its three ascending edges (if extended) would not meet at a single point. So here we have a sort of negative illusion: a drawing would not “match” a real photograph of any pyramid-base, yet it could match quite well an abstract description of a pyramid base — say, one that describes how its faces and edges fit together qualitatively, but not quantitatively.

Another topic concerns “camouflaged” figures. A figure “4” embedded within a grid of lines is not normally seen as such because, we presume, one describes the scene as a square and parallelogram. Study of this kind of concealment can tell us something about the “principles” according to which our visual system “usually” describes scenes as made up of objects. But once the “4” has been pointed out or discovered, it is then “seen” quite clearly! A good theory must also account for phenomena in which it is possible to change and elaborate one’s “image” of the same scene in ways that depend on changes in his interpretation and understanding of the structure “shown” in the picture.

A simpler and more interesting example of a figure with two competitive descriptions is the ordinary square! Young children know the square and the diamond as two quite distinct shapes, and the ambiguity persists in adults [Attneave, 1954]. Figures oriented along orthogonal axes are seen as squares, while those rotated 45 degrees are seen as diamonds. How can we explain this? Since the individual objects are in fact geometrically identical, the effect must have something to do with their arrangement. It is tempting to incant the phrase “the whole is more than the sum of the parts.”

Now consider a descriptive theory. If one is asked to describe this scene, he will say something like: “There are two rows, each with four objects. One is a horizontal row of… etc”. We ignore details here, but suggest that the description is dominated by the grouping into rows, as indicated by their priority in the verbal presentation of the description. In section 4.6 we discuss a program that does something of this sort.

DESCRIPTIONS AS ABSTRACT DATA STRUCTURES By “description” we do not usually mean “verbal description”; we mean an abstract data structure in which are represented features, relations, functions, references to processes, and other information. Besides representing things and relations between things, descriptions often contain information about the relative importance of features to one another, e.g., commitments about which features are to be regarded as essential and which are merely ornamental. For example, much of linguistic structure is concerned with the ability to embed hierarchies of detail into descriptions: subordinate clause formation and other word-order choices often reflect priorities and progressions of structural detail in the descriptions that are “meant.” We will return to this in section 5.

Once committed to describing a row of things, the choice between seeing squares and diamonds begins to make more sense. Which description does one choose? Apparently, the way one describes a square figure depends very much on how one chooses (in one’s mind) the axis of symmetry. Consider the differences in the figures’ descriptions in each of the two obvious choices of orientation:

DIAMOND ORIENTATION (Point-aligned)	SQUARE ORIENTATION (Side-aligned)
points on axis	sides parallel to axis
one point on each side	two points on each side
made of two triangles	made of two rectangles
unstable on ground / hurts when squeezed	stable flat bottom / safe to pick up

These two descriptions could hardly be more different! No wonder that most 3-year-olds do not believe that they are the same. In fact, children’s drawings of diamonds often come out as two triangles stacked together, indicating that their descriptive image is a composition of two triangles, or at least that the most important features are the points on the symmetry axes. Our mystery is then almost solved: whatever process set up the description in terms of rows set up also a spatial frame of reference for each group.

Since one has to choose an axis for each square and “other things being equal” there is no strong reason locally for either choice, one tends to use the axis inherited from the direction of its “row.” The fact that you can, if you want, choose to see any of the objects as either diamond or square only confirms this theoretical suggestion — the choice is by default only, and hence would be expected to carry little force.

Once this door is opened, it suggests that other choices one has to make in visual description also can depend on other alien elements in one’s thoughts — as well as on other things in the picture! Every simple figure is highly ambiguous. In a face, a circle can be an eye, a mouth, an ear, or the whole head. There should be no difficulty in admitting this to our theory or to the computer programs that demonstrate its consistency and performance. Traditional theories directed toward physical (rather than on computational, or symbolic) mechanisms were inherently unable to account for the influence of other knowledge and ideas upon “perception”.

2.2 Sensation, Perception and Cognition

Our discussion of how images depend on states of mind is part of a broader attack on the conventional view of the structure of mind. In today’s culture we grow up to believe that mental activity operates according to some scheme in which information is transformed through a sequence of stages like:

WORLD  –>  SENSATION  –>  PERCEPTION  –>  RECOGNITION  –>  COGNITION

Although it is hard to explain exactly what these stages or levels are, everyone comes to believe that they exist. The “new look” in ideas about thinking rejects the idea that there are separate activities like “perception” that precede and are basically independent of “higher” intellectual activities. What one “sees” depends very much on one’s current motives, intentions, memories, and acquired processes. We do not mean to say either that the old layer-cake scheme is entirely wrong or that it is useless. Rather, it represents an early concept that was once a clarification but is now a source of obscurity, for it is technically inadequate against the background of today’s more intricate and ambitious ideas about mechanisms.

ANATOMICAL VS. COMPUTATIONAL STRUCTURE The higher nervous system is embryologically and anatomically divided into stages of some sort and this might suggest a basis for the popular-science hierarchy. This makes sense for the most peripheral sensory and motor systems, in which transmission between anatomical stages is chiefly unidirectional. But (presumably) when we go further in the central direction this is no longer true, and one should not expect the geometrical parts of a cybernetic machine to correspond very well to its “computational parts.”

Indeed, the very concept of “part”, as in a machine, must be rebuilt when discussing programs and processes. For example, it is quite common in computer programs — and, we presume, in thought processes — to find that two different procedures use each other as subprocedures! We shall see this happening throughout section 5. In such a case one can hardly think of either process as a proper part of the other. So the traditional view of a mechanism as a HIERARCHY of parts, subassemblies and sub-sub-assemblies (e.g., the main bearing of the fuel pump of the pitch vernier rocket of the second ascent stage) must give way to a HETERARCHY of computational ingredients.

It is unfortunate that technical theories, and even practical guidelines, for such heterarchies are still in their infancies. The rest of this chapter discusses some aspects of this problem.

2.3 Parts and Wholes

A recurrent theme in the history of psychological thinking involves recognizing an important distinction without having the technical means to give it the appropriate degree of precision. Consequently, the dividing line becomes prematurely entrenched in the wrong place. An influential example was the concept of “Gestalt”. This word is used in attempts to differentiate between the simplest immediate and local effects of stimuli, and those effects that depend on a much more “global” influence of the whole stimulus “field”.

Consider a visual example: an architectural arch is in one sense “nothing but” three blocks. But the arch has properties as a single whole that are not inherited directly from properties of its parts in any simple way. Obviously the properties one has in mind do not reside in the individual building blocks, they “emerge” from the arrangements of those parts. And one finds this in even simpler situations. Obviously we react to a simple outline square in a way that is very different from our reactions to four separate lines.

The question “whence comes the square if not from its parts” is not really very serious here, for it is easy to make theories about how one might “perceive” a shape if there are enough easily-detected features to approximately delineate its geometric form. But there is no similarly easy solution to the kinds of problems that arise when one looks at three-dimensional scenes.

To clarify the mathematical distinction, consider local vs. global properties. Imagine innumerable experiments, in each of which we choose a different point of a picture to look at, and record what we see only within a very small circle around that point. It can be shown mathematically that certain properties (like whether a figure consists of solid rectangles) can be verified strictly by local micro-scene inspections. However, properties such as CONNECTEDNESS cannot be defined in terms of local properties alone. Connectedness requires global computational analysis.

The question “Is the whole more than the sum of its parts” is certainly provocative and insightful. But it must be recognized also as vague, relative, and metaphorical. What is meant by “parts” and, more important, what is meant by “sum”?

In the case of the rectangles a trivial sense of “sum” will suffice: not even adding up evidence is necessary, for we can make the decision in favor of rectangle, and let any single exception to our condition on the local “micro-scenes” have absolute veto power. So the “sum of the parts” is simply the agreement of all local evidence. For connectedness we seem to need something more complicated, computationally. We have studied this situation rather deeply in PERCEPTRONS: connectedness is a property that is quite important and very thoroughly understood in classical mathematics — it is in fact the central concern of the entire subject of Topology.

For example, here are several quite different-looking conditions each of which can be used to define the same concept of connectedness:

• PATH-CONNECTION: For any two black points of the picture, there is a path connecting them that lies entirely in black points.
• PATH-SEPARATION: There is no closed path, entirely in white points, such that there are some black points inside the path and some black points outside the path.
• SET-SEPARATION: The black points cannot be divided into two non-empty sets which are separated by a non-zero distance — that is, no pair of points, one from each set, are closer than a certain distance.
• TOTAL-CURVATURE: Assume that there are no “holes” in the black set — that is, white points that are cut off from the outside by a barrier of black points. Then compute the sum of all the boundary curvatures (direction-changes at all edges of the figure), taking convex curves as positive and concave curves as negative. The picture is connected if this sum is exactly 360 degrees. If it is a multiple of 360, this gives the number of objects!

Each of these suggests different computational approaches. Depending upon what resources are available, one or another will be more efficient, use more or less memory, time, hardware, etc. Each definition involves very large calculations in any case, except the fourth, in which one computes simply a sum of what one observes in each small neighborhood. However, the fourth definition does not work in general, but only for figures without holes. And, to be sure that condition is satisfied one must have another source of information or else the definition is somewhat circular, because to be able to see that there are no holes is really equivalent to being able to see that the background is connected!

We know exactly what it means for the number seven to be the sum of the numbers three and four. But when we ask whether a house is just the sum of its bricks, we are in a more complicated situation. One might answer: “Yes, there is nothing but bricks there”. But another kind of answer could be: “No, for the same bricks arranged differently would have made a very different house.”

The answer must depend on the purpose of the question. If we admit only “yes” or “no”, there is no room for refinement and subtlety of discussion. We do not really want either of the answers “Yes, it is nothing but the sum” or “No, it is a Gestalt, a totally different and new thing”. We really want to know exactly how the response, image, or interpretation of the situation is produced: we want an explanation of the phenomenon. And the terms of the explanation must be appropriate to the kind of technical question we have in mind.

Thus one might ask, about some aspect of a person’s behavior:

• COMPONENTS: Can the phenomenon be produced in a certain kind of theoretical neural network?
• LEARNING: Can it be learned by a certain kind of reinforcement schedule according to certain proposed laws of conditioning?
• COMPUTATIONAL STRUCTURE: Can this result be computed by a computer-like system subject to certain restrictions, say, on the amount of memory, or on the exclusion of certain kinds of loops interconnecting its components?
• COMPUTATIONAL SCHEMATA: Can the outer behavior of this individual reasonably be imitated by a program containing such-and-such a data-structure and such-and-such a syntactic analyser and synthesizer?

The way in which the whole depends upon its parts, for any phenomenon, has a direct bearing on how such questions can be answered. But to supply sensible answers, one needs a stock of crisp, precise ideas about how parts and wholes may be related!

It is important to recognize that these kinds of problems are not special to Psychology. Water has properties that are not properties either of hydrogen or oxygen, yet chemistry is no longer plagued by fights between two camps say, “Atomist” vs. “Gestalt”. This is not at all because the problem is unimportant: exactly the opposite! The reason there are no longer two camps in Chemistry is because all workers recognize that the central problems of the field lie in developing good theories of the different kinds of interactions involved, and that the solution of such problems lie in constructing adequate scientific and mathematical models rather than in defending romantic but irrelevant philosophical overviews. But in Psychology and Biology, there remains a widespread belief that there are phenomena of mind or of cell that are not “reducible” to properties and interactions of the parts. They are saying, in essence, that there can be no adequate theory of the interactions.

Consider a concrete engineering example: a supported rod vs. a triangular truss. It is relatively easy to bend a thin rod, but much harder to bend a structure made of several such rods. Where does the extra stiffness come from? The answer is that the “new property” is indeed inherited from the parts, because of the arrangement, but in a peculiar way. In the truss, a force at the middle is resisted — not by bending-forces across the rods — but by compression and tension forces along the rods. The resistance of a thin rod to forces along it is much greater than the resistance to forces across it. So the increased strength is indeed “reduced”, in the Theory of Static Mechanics, to the interactions of stresses between members of the structure. Even the properties of a single rod itself can be explained in terms of more microscopic interactions of the tensile and compressive forces between its own parts, when it is strained. Thus one obtains such a beautiful and accurate model that there remains no mysterious “Gestalt” problem at all.

Of course, our real concern is with problems of intelligence, rather than with engineering mechanics. But many problems that seem at first to be “purely psychological” often turn out to center around just such problems of wholes and parts. And with such an interpretation, we may replace an elusively ill-defined psychological puzzle by a much sharper problem within the theory of computation.

The computer is the example par excellence of mechanisms in which one gets complex results from simple interactions of simple components. In asking how thought-like activity could be embedded in computer programs, scientists for the first time really came to grips with understanding how intelligent behavior could be made to emerge from simple interactions.

To illustrate extreme versions of atomism vs. gestaltism one might consider these caricatures:

• Extreme ATOMISM: All behavior can be understood in terms of simple functions of neural paths that run from single receptors, through internuncials, to effectors.
• Extreme GESTALTISM: The essence is in the whole pattern. Many simple examples show that the response is made to the whole stimulus and cannot be represented as simple sums or products of simple local stimulations.

Clearly one does not want to set a threshold between these; one wants to classify intermediate varieties of interactions that might be involved, arranged if possible in some natural order of complexity.

Thus in PERCEPTRONS we studied a variety of simple schemas such as these:

• EXTREMELY ATOMIC ALGORITHM: One of the input wires is connected to the output, the others to nothing.
• VETO ALGORITHM: If every input says “yes”, the output is “yes”. If any input says “no”, the output is “no”.
• MAJORITY ALGORITHM: If M or more of N inputs say “yes”, output is “yes”.
• LINEAR SUM ALGORITHM: To each input is assigned a “weight”. Add together the weights for just those inputs that say “yes”. The output is just this sum.
• LINEAR THRESHOLD ALGORITHM: Use the LINEAR SUM algorithm, except make the output “yes” if the sum is greater than a certain “threshold”, otherwise the output is “no”. (Exercise: the reader should convince himself that “extremely atomic”, “veto”, and “majority” are special cases of “linear threshold”.)
• EQUIVALENT-PAIR ALGORITHM: The input is considered to be grouped in pairs. The output is “yes” only when, for every pair, the two members have the same input value. (The reader should convince himself that this is not a special case of “linear threshold”!)
• SYMMETRICAL ALGORITHM: The response is “yes” if the pattern of inputs is symmetrical about some particular center, or about some particular linear axis.
• PERCEPTRON ALGORITHM: First some computationally very simple functions of the inputs are computed, then one applies a linear threshold algorithm to the values of these functions.
• GAMBA PERCEPTRON: A number of linear threshold systems have their outputs connected to the inputs of a linear threshold system. Thus we have a linear threshold function of many linear threshold functions.

The community of ideas in the area of computer science makes a real change in the range of available concepts. Before this, we had too feeble a family of concepts to support effective theories of intelligence, learning, and development. Neither the finite-state and stimulus-response catalogs of the Behaviorists, the hydraulic and economic analogies of the Freudians, nor the holistic insights of the Gestaltists supplied enough technical ingredients to develop such an intricate subject.

Computer science has brought a flood of such ideas, well defined and experimentally implemented, for thinking about thinking — only a fraction of them have distinguishable representations in traditional psychology:

• symbol table	• pure procedure
• time-sharing	• calling sequence
• functional argument	• memory protection
• dispatch table	• error message
• function-call trace	• breakpoint
• formal language	• compiler
• indirect address	• macro language
• property list	• data type
• hash coding	• micro-program
• format matching	• syntax-direction
• closed subroutine	• pushdown list
• interrupt	• communication cell
• common storage	• decision tree
• hardware-software trade-off	• serial-parallel trade-off
• time-memory trade-off	• conditional breakpoint
• asynchronous processing	• interpreter
• garbage collection	• list structure
• block structure	• look-ahead
• look-behind (cache)	• diagnostic program
• executive program	• operating system

3.0 Analysis of Visual Scenes

Before discussing scene-analysis in detail, we have a few remarks about the nature of problems in this area. In the early days of cybernetics [McCulloch-Pitts 1943, Wiener 1949] it was felt that the hardest problems in apprehending a visual scene were concerned with questions like “why do things look the same when seen from different viewpoints”, when their optical images have different sizes and positions.

How does one capture the “abstraction” or “concept” common to all the particular examples. For two-dimensional character-recognition, this kind of problem is usually handled by a two-step process in which the image is first “normalized” to standard position and then “matched” by a correlation or filtering process to one of a set of standard representatives. In practical engineering applications, the “normalizing” often failed because it could not disarticulate parts of images that touch together, and “matching” often failed because it is hard to make correlation-like processes attend to “important” parts of the figures instead of to ornaments. Even so, such methods work well enough for reasonably standardized symbols.

If, however, one wants the machine to read the full variety of typography that a literate person can, the problem is harder, and if one wants to deal with hand-printing, quite different methods are needed. One is absolutely forced to use exterior knowledge involving the pictures’ contexts, in situations like:

CONTEXT IN RECOGNITION THE CAT Here the distinction between the ‘H’ and the ‘A’ (written with identical central glyphs) is not geometric at all, but exists only in one’s knowledge about the language [Selfridge, 1955; Bledsoe and Browning, 1959].

But we will not stop to review the field of character-recognition, for its technology is quite alien to the problems of three-dimensional scenes. This is because the problems that concern us most, like how to separate objects that overlap, or how to recognize objects that are partially hidden (either by other objects or by occluding parts of their own surfaces), simply do not occur at all in the two-dimensional case.

Some more interesting two-dimensional problems require description when geometric matching fails; a conceptual “A” is not simply a particular geometric shape; it is:

STRUCTURAL DEFINITION OF ‘A’ “Two lines of comparable length that meet at an acute angle, connected near their middles by a third line.”

3.1 Programs for Finding Bodies in Scenes

Let us review quickly how Guzman’s SEE program works. First a collection of “lower level” programs are made to operate directly on the optical data. Their job is to find geometric features of the picture — regions, edges and vertices — so that the scene can be described in a simple way in the program’s data-structure. Next, the vertices are classified into “types”. The most important kinds are these:

• ARROW: Usually caused by an exterior corner of an object, where two of its plane surfaces form an edge. Inserts a link between the two regions bounded by the smaller angles.
• FORK: Usually due to three planes meeting at a convex corner of an object; causes three links between those regions.
• TEE: Usually caused by an edge of one object disappearing under an edge of another object (occlusion).
• ELL & TRANS: Corner boundary and transition configurations.

Using these clues, and representing the resulting relations by simple abstract networks, many scenes are “correctly” analyzed into objects.

If two TEE vertices have their stems in the same line then we create two more links. This often does just the right thing for an object whose picture is divided into two separate parts by another object in front.

Many scenes are handled correctly by just these simple rules, but many are not. For example, the basic assumption about the FORK linking its three regions is not true of concave corners, and the “matching TEE” assumption may be false by coincidence, so that “false links” may be produced.

Guzman introduced several methods for correcting such errors. One method involves a conservative procedure in which groupings are considered to have different qualities of connectedness. Two high-quality groups that are connected together by only a single link are broken apart — the link is deleted.

A second error-correction method is more interesting. Here we observe that the TEE vertex really has a special character, quite opposed to that of the FORK and the ARROW. The most usual physical cause of a TEE is that an edge of one object has disappeared under an edge of another object. Hence we should regard the TEE joint as evidence against linking the corresponding regions! Guzman’s implementation of this was to recognize certain kinds of configurations as special situations in which the existence of one kind of vertex-type causes inhibition or cancellation of a link that would otherwise be produced by the other vertex-type.

This technique corrects many errors that the more “naive” system makes, especially in objects with concavities. Note that it attempts to compute Connectedness by less local methods because of the effects of vertex-types of contiguous or closely-related geometric features.

It has been pointed out by D. Huffman [1970] that each line in a line-drawing may be interpreted as a physical edge formed (we assume) by the intersection of two planes, at least locally. In some cases one can see parts of both planes, but in other cases only one. A T-joint is good evidence that the edge involved is of the latter kind, and once one assigns such an interpretation to an edge, then it follows immediately that the adjacent Guzman links to the alien surface ought to be rejected. Accordingly, Huffman developed a number of procedures for making detailed global interpretations from local edge-region assignments.

But SEE has faults, among which are:

• INFLEXIBILITY: If its very first proposal is not acceptable, the body-aggregation program ought to be able to respond to complaints from other higher and lower level programs and thus generate alternative “parsings” of the scene.
• ORDINARY “MISTAKES”: Certain simple figures are not handled correctly. All figures are inherently ambiguous, but SEE sometimes links all regions of distinct overlapping objects together.
• IGNORANCE: It has no way to use knowledge about common or plausible shapes. While it is a virtue to be able to go so far without using such exterior information, it is a fault to insist on this!

Following Guzman’s work, Martin Rattner described a procedure, called SEEMORE, that can handle some of these problems [Rattner 1970]. While it uses linking heuristics much as did Guzman, SEEMORE puts more emphasis on local evidence that an edge might separate two bodies. These “splitting heuristics” operate initially at certain kinds of vertices, notably TEE-vertices and vertices with more than three edges. When there is more than one plausible alternative, SEEMORE uses other evidence to make tentative choices of how to continue a splitting line, but stores these choices on back-up lists that can later be used to generate alternative parsings.

Chapter 4. Description and Learning

The concepts we used to analyse ANALOGY and SEEING are just as vital in understanding LEARNING. It was traditional to try to account for learning in terms of such primitives as “conditioned reflex” or “stimulus-response bond”. The phenomena of learning become much more intelligible when seen in terms of “description” and “procedure”.

As we develop this idea, we find ourselves forced to question the whole tradition in which one distinguishes a special sub-set of mental or behavioral processes called “learning”. Nothing but disaster can come from looking for three separate theories to explain (for example):

• How one learns mathematics,
• How one thinks mathematically once he has learned to, and
• What mathematics is, anyway.

In this chapter we shall argue that many problems about “learning” really are concerned with the problem of finding a description that satisfies some goal. Everyone would agree that getting to know one’s way around a city is “learning”. Similarly, we see solving a problem often as getting to know one’s way around a “micro-world” in which the problem exists.

Think, for example, of what it is like to work on a chess problem. Here the microworld consists of the network of situations on the chessboards that arise when one moves the pieces. Solving the chess problem consists largely of getting to know the relations between the pieces, and how the moves affect things. As exploring goes on, a grouping first seen as three pieces playing different roles is now described in terms of a single relation between the three, such as “pin”, “fork”, or “defense.”

In our laboratory the chess program written by R. Greenblatt plays fairly good chess, by amateur tournament standards. But visitors are always disappointed to find that this program does not “learn”, in the sense that it carries no permanent change away from the games it plays. We claim that there is indeed an important kind of learning within the program: this is in the position-description summaries that are constructed and used as it analyses the positions it is playing. But because board positions do not often repeat exactly in subsequent games, there would be no point in keeping such records permanently until higher-level strategy-oriented descriptions are developed.

We think that learning to learn is very much like debugging complex computer programs. To be good at it requires one to know a lot about describing processes and manipulating such descriptions. We classify various not-entirely-separable ideas of learning as follows:

• Learning by development or maturation
• Learning without description (by quantitative adaptation)
• Learning by building and modifying descriptions
• Learning by being taught / Learning by Analogy / Learning by being told
• Learning by Understanding

4.1 An Example of Learning: Piaget’s Conservation Experiments

A classical experiment of Jean Piaget shows remarkably repeatable patterns of response of children (in the age range of 4-7 years) to questions about quantity.

When presented with two equal rows of eggs and egg-cups, a child agrees they are equal. But if one row is spread out spatially, a typical 5-year-old asserts there are “more eggs”, whereas a typical 7-year-old replies “Of course not!”

Similarly, in liquid conservation experiments, water poured from a wide jar into a tall, thin jar is declared by younger children to be “more water”, while older children insist it is the same amount because no water was added or removed.

How can we explain the difference between the less and more mature children? We see two problems here from the point of view of learning. First, how is the pre-conservationist view acquired (and executed); then how is it replaced by a conservationist one? To many psychologists only the second seems interesting, assuming the child is simply ‘dominated by perception’. Our view is that the child chooses to describe the situation in terms of vertical extent.

When asked for a quantitative comparison, the child converts a complex dual description (“tall thin column vs. short wide column”) into a simplified single-descriptor comparison (“high column vs. low column”). Once this description is selected, any decision process chooses the higher one.

The transition to conservationist thinking occurs when the child develops higher-order knowledge to classify and prioritize descriptors — recognizing that HISTORICAL/IDENTITY rules (“nothing added or taken away”) take precedence over SPATIAL extent heuristics.

4.3 – 4.4 Learning by Building Descriptions

We can illustrate powerful concepts of structural learning in the context of a procedure developed by P. Winston. When presented with a scene of toy blocks, Winston’s program observes relations between features and regions, groups these to find proposed objects, and builds a descriptive network of relations (e.g., SUPPORTED-BY, IN-FRONT-OF, CLEARTOP).

When taught the concept of an ARCH:

• Step 1: Presented with a valid arch (SCENE 1), it saves the full relational network.
• Step 2: Presented with a near-miss where two uprights touch (SCENE 2) and told ‘NOT AN ARCH’, it computes the difference description and adds a MUST-NOT-TOUCH restriction.
• Step 3: Presented with a near-miss where the top lintel is absent (SCENE 3) and told ‘NOT AN ARCH’, it adds a MUST-BE-SUPPORTED-BY enforcement pointer.
• Step 4: Presented with a valid arch with a triangular roof wedge (SCENE 4), it finds the smallest common generalization class in memory (‘PRISM’) that encompasses both bricks and wedges.

LEARNED CONCEPT DEFINITION An ARCH is defined by the learned program as: “A structure in which a prismatic body is supported by two upright blocks that do not touch one another.”

5.0 – 5.3 Knowledge, Generality, and the Blocks World

An intelligent person is vastly more versatile than special-purpose programs. To build programs that need not be rebuilt for every minor change in task, the system must possess ‘common sense’ knowledge about time, space, causes, goals, and actions.

We contrast two research strategies in Artificial Intelligence:

• The POWER Strategy: Seeks generalized computational power, faster hardware, or uniform theorem-proving resolution algorithms independent of the specific domain.
• The KNOWLEDGE Strategy: Seeks progress through better ways to express, organize, recognize, and execute diverse and domain-specific structures of knowledge.

Terry Winograd’s BLOCKS WORLD (SHRDLU) exemplifies the Knowledge strategy. Operating within a micro-world consisting of a simulated robot arm, a table, a box, and various blocks/pyramids, the system integrates syntactic parsing, semantic interpretation, and deductive planning (using Hewitt’s PLANNER language).

When commanded to ‘PUT B3 ON B4’, the system calls procedural theorems like tc-PUTON, which recursively generates subgoals: clearing the top of B3 (tc-CLEARTOP), finding space on B4 (tc-FINDSPACE), grasping B3 (tc-GRASP), and moving the arm (tc-MOVEHAND). The system maintains a complete subgoal tree and event history, allowing it to answer questions not only about the physical state of the world but about its own past actions and intentions (e.g., ‘Why did you pick up that block?’).