Insight Editions · No. 285
Artificial Intelligence: The Very Idea
John Haugeland
Haugeland argues that Artificial Intelligence rests on one daring theoretical idea: that thinking is computation, so that we are, at root, computers ourselves.
A study guide to Artificial Intelligence: The Very Idea by John Haugeland. Shared free for personal reading; not for sale and not endorsed by the authors or publishers. Please read and buy the original.
The Book on One Page
John Haugeland's Artificial Intelligence: The Very Idea (1985) is a philosopher's account of what AI actually claims: not that machines can fake intelligence, but that thinking is computation, so that we are, at root, computers ourselves. He traces the idea from Hobbes to the stored-program computer, defines a computer exactly as an interpreted automatic formal system, coins the name GOFAI (Good Old Fashioned Artificial Intelligence), and then tests the programme against its real achievements and against real people. His verdict is a careful skeptic's: the very idea is neither preposterous nor inevitable, a genuine empirical hypothesis whose hard questions were still open after thirty years.
Core claim: AI rests on the hypothesis that intelligence just is rational, quasi-linguistic symbol manipulation; computers show that such manipulation can be fully mechanical and still meaningful, which dissolves the old paradox of mechanical reason, but whether real minds work this way is an open empirical question.
Scope: Philosophy of mind from Copernicus, Galileo, Hobbes, Descartes and Hume; the theory of formal systems, interpretation and computation; five computer architectures; and AI research from 1955 to 1985 (Logic Theorist, GPS, SHRDLU, frames and scripts, the frame problem), weighed against pragmatics, imagery, feeling and ego.
Key figures: Hobbes ("ratiocination is computation"), Turing (universal machines and the test), Newell, Shaw and Simon (heuristic search), Winograd (SHRDLU), Minsky and Schank (frames and scripts); the 1957 forecast of a chess champion within ten years, delivered thirty years late by Deep Blue (1997).
Best for: Anyone who builds or governs AI and wants the conceptual foundations stated precisely: what makes a symbol mean something, why architectures matter, why micro-world demos mislead, and why common sense, context and relevance are the hard part. It reads as a map of where today's LLM debates came from.
The modern mind was invented in the seventeenth century. Copernicus split appearance from reality; Galileo showed that a notation can represent anything it can be used to reason about; Hobbes declared that thinking is computation on inner symbols; Descartes made the divorce of thought from thing final; Hume built a mental mechanics with no meaning left in it. The result was the paradox of mechanical reason: if a process attends to meanings it cannot be mechanical, and if it is mechanical it cannot be reasoning. Behind it sits the mystery of original meaning: why thoughts mean something by themselves while words and books only borrow meaning.
Computers are the proposed way out. A formal system is a self-contained, digital token game, playable by finite players; by the Automation Principle, any system whose moves are fixed by algorithms can be built as a machine, and fallible heuristics can be automated by redescribing an algorithm as an estimator. Add an interpretation that makes the machine's moves make reasonable sense, and you have a computer. The Formalists' Motto ("take care of the syntax, and the semantics will take care of itself") explains how mechanical token shuffling can be reasoning under a second description, and homuncular analysis breaks intelligence into ever dumber, finally mindless components.
GOFAI is a definite, falsifiable thesis. It holds that intelligence is due to reasonable thinking, and thinking is internal automatic symbol manipulation. That excludes mere simulation (a hurricane model is not a hurricane) and any rival theory of mind. Because computers can differ radically in memory and control (Babbage, Turing, von Neumann, LISP, production systems) while all being universal, the mind's own architecture is an empirical variable.
The research record is instructive more than triumphant. Cybernetics and statistical machine translation failed for lack of understanding. Heuristic search tamed the combinatorial explosion but left the intelligence in formulating the problem. Micro-worlds such as SHRDLU dazzled only because their domains were stripped of everything that needs wit: the world cannot be cut into independent fragments. Stereotypes (frames, scripts) organised common sense brilliantly but stumbled on selection, context and relevance, and the frame problem showed how hard it is to keep beliefs about a changing world up to date.
Real people set a higher bar. Intelligence must make pragmatic sense (Grice's conversational implicature), may rely on quasi-pictorial imagery that does not fit symbol manipulation, involves feelings that resist clean segregation (moods above all), and may require ego involvement: understanding a story can depend on feeling embarrassment on someone's behalf. Each could be walled off by a segregation strategy, or could prove an alien intruder in GOFAI's home territory.
The conclusion is skepticism, not debunking. A priori proofs that AI is impossible rest on thin premises, consciousness included, and Simon and Newell's failed 1957 forecasts show misjudged research, not a refuted theory. But history has imposter paradigms, behaviourism among them, with every institutional sign of success. Haugeland is not convinced GOFAI is impossible, and far from persuaded it is inevitable. Forty years later his skepticism about GOFAI looks right, while the broader idea of thinking machines has revived in learned, non-symbolic form that his definition excludes.
| Ch | Title | One-line payload |
|---|---|---|
| 0 | Introduction | AI's goal is genuine minds; Turing's test is a fair criterion; "computers only do what they're programmed to do" fails on levels of description. |
| 1 | The Saga of the Modern Mind | From Copernicus to Hume, thought becomes symbol manipulation, leaving the paradox of mechanical reason and the mystery of original meaning. |
| 2 | Automatic Formal Systems | Digital, medium-independent, finitely playable token games can be automated by algorithms, and heuristics by redescription. |
| 3 | Semantics | Interpretation is coherence; a computer is an interpreted automatic formal system; GOFAI is named and the paradox dissolved by two descriptions of one process. |
| 4 | Computer Architecture | Babbage, Turing, von Neumann, LISP and production systems are all universal yet radically different, so mental architecture is an empirical variable. |
| 5 | Real Machines | Search, micro-worlds, stereotypes and the frame problem show that the hard part of intelligence is common sense, context and relevance. |
| 6 | Real People | Pragmatic sense, imagery, feelings and ego may resist segregation from cognition; the 1957 forecasts failed; GOFAI is neither impossible nor inevitable. |
| S | Master Takeaways | The book's insights, frameworks, critical lens and links to AI practice. |
Introduction
Artificial Intelligence, Haugeland says, is not after a clever fake of intelligence but "machines with minds, in the full and literal sense", and it rests on one daring theoretical idea: we are, at root, computers ourselves. The book's subject is that "very idea", that thinking and computing are radically the same. The Introduction sets the tone (neither scoffer nor booster), explains why computers rather than clockworks earned scientific respect, defends Turing's test, and dismantles the slogan that a computer "can only do what it's programmed to do."
Minds, Artificial and Natural
Philosophers have asked what minds are and what sets people apart for millennia with scant progress by scientific standards, until a recent "sudden and brilliant flowering" in the philosophy and science of mind. The epitome of that revolution is Artificial Intelligence, "the exciting new effort to make computers think". Its goal is the genuine article. In a note Haugeland suggests the field might better be called "Synthetic Intelligence": artificial diamonds are fakes, while synthetic diamonds are real diamonds that happen to be manufactured, and AI wants the synthetic kind.
Popular opinion splits into scoffers (the idea is as ridiculous as a car that really hates you, or jailing the bullet in a murder) and boosters (machine minds are as inevitable as interplanetary travel). Both thump the table. Haugeland's stance is that Artificial Intelligence is neither preposterous nor inevitable: it rests on a powerful idea that might well be right, or right in some respects, and might just as well not be.
- Explain. Say clearly and with an open mind what AI is really about.
- Credential. Exhibit the philosophical and scientific credentials behind its enormous appeal.
- Assess. Look at what AI has actually accomplished, which requires an abstract account of what computers are, the metaphysics of "meaning" in a material universe, and the common-sense confusions about language, knowledge, personality and common sense itself.
Fiction, Technology and Theory
Science fiction already had two genres of intelligent artifact. The "creature feature" gives us monsters and androids, man-made but basically like animals (the creations of Hephaestus and Dr Frankenstein, anthropoid slaves distinguishable only by serial numbers). The robot genre gives us clanking contraptions with springs for flesh and wires for nerves, each generation an extrapolation of the latest marvel: clockwork gears, steam engines, automatic looms, hydraulics, telephone switchboards. AI descends from the robot line; it belongs "at IBM, not at du Pont or Genentech."
The difference is that no respectable scientist tried to build an intelligent switchboard, while research on intelligent computers is "big time". The reason is theoretical, not technological. A central Western tradition holds that thinking is rational manipulation of mental symbols (ideas). Clocks do nothing like that. Computers can manipulate arbitrary "tokens" in any specifiable manner, so if the tokens are arranged to be symbols and the manipulations to be rational, the machine should think. If the traditional theory is true, such a machine has a mind of its own.
Abstraction and medium independence
Calling something a symbol or a manipulation is an abstract characterisation: inessential details are omitted, which is different from being vague. Haugeland's analogy is two specifications for an appliance motor. One engineer gives every part's shape and material (concrete); the other gives only minimum horsepower, available space and noise level (abstract), and the motor could be metal or plastic, round or square, built on any physical principle, and still meet the specification exactly.
On the symbol-manipulation theory, intelligence depends only on a system's organisation and functioning as a symbol manipulator, which is even more abstract than horsepower. So whether the substrate is electronic, physiological, hydraulic or fibre-optic is beside the point; electronics matter only because they are currently the cheapest way to build flexible symbol manipulators. A deeper consequence follows: if AI is about abstract principles of mental organisation, then AI, psychology and philosophy of mind melt into one subject, studied with different tools. Enthusiasts call the merged field cognitive science, with AI, psychology and parts of philosophy, linguistics and anthropology as "subspecialties" of the study of symbol manipulation.
Scope
- Only cognitive-science AI. Haugeland ignores commercial ventures such as "expert systems" that make no pretence of applying psychological principles.
- No alien intellects. He also sets aside whether computers might have some inhuman kind of intellect. His hunch is that "human chauvinism" is built into our very concept of intelligence: escape the prejudice and we would no longer know what we were talking about.
- Materialism assumed. A note concedes that if thoughts live in immaterial souls, AI is ruled out from the start; for discussion, the book assumes intelligence is (or could be) realised in matter.
The only theoretical reason to take AI more seriously than clockwork fiction is the suggestion that our own minds work on computational principles. AI matters as part of the theory that people are computers.
What Is Intelligence? The Turing Test
Very little turns on defining intelligence, because Alan Turing (1950) offered a criterion that "satisfies nearly everyone". Turing was annoyed by fruitless disputes over word meanings and proposed replacing the question with a test, predicting that by the year 2000 machines would pass a modest version and contrary "definitions" would quietly fade away.
Three mutual strangers play. Two witnesses of opposite sex answer questions by teletype; the interrogator tries to guess which is which. The man tries to fool the interrogator by pretending to be the woman; the woman tries to help. Turing's twist is to substitute a computer for the man: the machine passes if, against average women, it fools the average interrogator as often as the average man can. (A note observes that Turing never says whether the interrogator is told a computer is present, which would change the questioning; Haugeland thinks the ambiguity does not matter.)
The teletypes and deception are window dressing to make the test experimental. The crux is talk: does the machine say the sorts of things people say in similar situations? Talk qualifies for two reasons. Turing notes that question and answer can introduce almost any field of human endeavour. More important, to converse beyond the superficial you must know what you are talking about: understanding the words is not enough, you must understand the topic. Turing's sample sonnet viva shows it. Asked whether "a spring day" would do as well as "a summer's day", the witness says "It wouldn't scan"; offered "a winter's day", it replies that nobody wants to be compared to one; pressed that Christmas is a winter's day and Mr Pickwick would not mind, it answers that "a winter's day" means a typical one, not a special one. The witness shows command of English, poetry, the seasons and people's feelings, all by talking, and the same could be done for politics, fine wines, electrical engineering or philosophy.
The test is also convenient: typing at a terminal is how one uses a computer anyway, so by accepting the test in spirit, if not the letter, researchers can concentrate on the cognitive question, namely what internal structure lets a system "say the right thing at the right time", and get on with computational psychology.
- IQ presupposes intelligence. IQ tests measure degree of intelligence on the assumption that there is some to measure; for computers, whether there is any at all is the issue.
- IQ is built for people. Tests exploit indicators valid only in the human case. A skill such as solving logic problems or comparing line diagrams may correlate with human smartness without requiring intelligence, and a manifestly unintelligent device could ace it by coincidence or gimmick. Pocket calculators extract square roots that many people have forgotten how to compute; they are not more intelligent. What is needed is a test of whether an entity is intelligent at all, in the human sense.
LLMs and the Turing test. Turing's own 1950 forecast was that around 2000 an average interrogator would have no better than a 70% chance of a correct identification after five minutes of questioning; that date passed without a convincing pass. Large language models changed the picture. In a 2025 three-party Turing test run by Cameron Jones and Benjamin Bergen at UC San Diego (a preprint at the time of reporting), GPT-4.5, prompted to adopt a humanlike persona, was judged to be the human 73% of the time, more often than the real human participants it was paired against; a persona-prompted LLaMa-3.1-405B came in near chance, while ELIZA and GPT-4o without the persona prompt were judged human far less often. The result revived Haugeland's own question from a new angle: fluent, topic-spanning talk now exists, and the dispute has moved to whether such talk shows understanding, or whether the test measured humanlikeness and persona rather than intelligence.
"They Can Only Do What They're Programmed to Do"
Doubts about automating creativity and freedom usually rest on this slogan. Haugeland separates two senses of "programmed" and shows that neither delivers the conclusion.
| Sense of "programmed" | What it means | Why the objection fails |
|---|---|---|
| Internal specification | A program is a careful specification of all relevant internal processes | The same holds for brains under laws of neuropsychology; it ignores levels of description, since "creative" and "free" apply only to the whole system |
| Overall design | "Programmed to keep the payroll" or "to find the best flight path" describes intended capacities | It is not clear design excludes creativity; humans are designed by evolution, so "designed for creativity" is no contradiction |
| Designed by a real designer | Evolution is a mindless process; computers have actual programmers, who get the credit | Creativity should depend on manifest competence, not ancestry; credit goes to designers only if outputs were prestored for playback |
Levels of description
The first sense ignores organisational levels. A stereo can be described as a device for reproducing recorded music, as a tangle of electronic components, or as a cloud of subatomic particles. No component, let alone particle, is "high fidelity"; that predicate exists only at the music-reproduction level. Likewise no individual brain function or machine operation is "creative" or "free"; such words belong to the level of the system or person as a whole. A note credits Dan Dennett with pressing this point.
Design and ancestry
If being designed for creativity were contradictory, we would be in trouble too: when healthy we "only do what we're designed to do", so we must be designed for creativity. The last resort says evolution only metaphorically designs, while computers have literal programmers. Haugeland asks why potential for inventiveness should depend on ancestry, like a hereditary title, rather than manifest competence.
The laboratory accident and the Exxon humans. Suppose the very same computer system arose from an incredible laboratory accident rather than a programmer: would that change whether it is creative? Conversely, suppose you were concocted from petroleum by-products at Exxon: would your later inventions and artworks belong to a team of chemists? Haugeland's answer is no in both cases. The experiment shows that origin is the wrong criterion; what matters is whether the inventions were dreamt up in advance and stored for playback, which is not how AI works.
What gets programmed is general information and principles, much as teachers instil them in pupils, and what the system does with them is not predictable by the designer. The striking example in 1985 was chess machines that outplay their programmers with moves the programmers would never have found. That is unsurprising if invention is often a rearrangement of previously available materials.
None of this proves machines can be creative or free. It shows that intuitions to the contrary are untrustworthy: "Nobody knows." The question awaits hard research, and the real issue remains whether, in the appropriate abstract sense, we are computers ourselves.
- The tempting view. Learning can look to psychology what energy is to physics or reproduction to biology, the defining topic, and a system with a child's "raw aptitude" could educate itself from experience and books and spare AI from codifying mature common sense.
- The hidden assumption. Framing learning as "given a knower, make it acquire" assumes knowing is easy and acquiring is hard. AI found the reverse: knowledge itself is extraordinarily complex, and even the general structure of a system with common sense is unclear.
- The consequence. Without knowing what a learner must acquire, the acquisition project cannot get off the ground, so AI must first understand knowledge and then tackle learning. Learning is essential to full intelligence but is not the most basic problem or a shortcut.
- The pedigree. A note quotes John McCarthy (1959): for a program to learn something, "it must first be capable of being told it", and cites Roger Schank (1983) pleading that learning be revived as a central topic.
The order was reversed. Within three decades the field did exactly what this box advised against. Statistical machine learning, then deep learning (image recognition breakthroughs from 2012) and the transformer architecture (2017) made learning from data the main route to competence, and large language models acquire broad, if imperfect, common-sense knowledge by training on text rather than having it codified by hand. Chess vindicated the box's other point in an unexpected way: Deep Blue beat world champion Garry Kasparov in 1997 largely by search, and AlphaZero (2017) learned chess, shogi and Go from self-play alone, producing play its designers could not have scripted. Haugeland's diagnosis survives in a different form: what learned systems actually know, and how it is structured, is now studied after the fact by interpretability research instead of being designed in first.
Out-of-date scope. Haugeland's decision to ignore expert systems and learning was reasonable for cognitive-science AI in 1985, but it frames AI almost entirely as explicit symbol manipulation. Most AI built since does not fit that frame, so his "very idea" now describes one historical research programme (he later names it GOFAI) rather than the field. His central question, whether computation of any kind amounts to thinking, carries over intact.
For an enterprise architect, Haugeland's levels-of-description point is a practical tool: properties such as "explainable", "fair" or "creative" belong to the system level of an AI service, not to individual weights, rules or API calls. Governance that looks only at components will miss them, just as no transistor in a stereo is high fidelity.
The Saga of the Modern Mind
Our common-sense concept of "the mind" was invented in the seventeenth century, alongside modern science and mathematics, and cognitive science is only the latest theory in that series. Haugeland tells the story as a drama: Copernicus splits appearance from reality, Galileo shows that a notation can represent anything it can be used to reason about, Hobbes declares that "ratiocination is computation", Descartes divorces thought from thing, and Hume builds a mental mechanics. The plot ends on a cliffhanger, the paradox of mechanical reason, which AI claims to resolve.
Haugeland confesses up front that he streamlines the history: "the medievals" are treated as if all alike, and ideas he pins on Galileo or Descartes were in fact common currency. What matters is that the larger drama is about right.
Copernicus and the End of the Middle Ages
The medieval world view was a Christian adaptation of Plato and Aristotle, with God as creator and cause of everything else. Before creating, God needed ideas (plans), and ordinary objects were more or less corrupt materialisations of those perfect ideas. Human minds had ideas too, and two stories explained how a thought could be "true of" a thing.
The Platonic triangle. Thought and thing form the base of a triangle with God's intellect at the apex. Human ideas are true insofar as they copy God's ideas; objects are "true" insofar as they conform to the same divine blueprints. Thought reaches thing only by a detour through the apex.
The Aristotelian base. Skip the apex and posit a direct relation: ideas are like pictures that resemble their objects and stand for them because they resemble them. Mind and world share the same form, realised in different substances. This resemblance theory was intuitive and unrivalled, until modern science sabotaged it.
Medieval cosmology was also Aristotelian. Earth sat at the centre, wrapped in rotating spheres of visible heavens and finally God's motionless heaven. The sensible world had five elements (earth, water, air, fire and the quintessence), each with a "natural place" it tended toward. The heavens were pure quintessence and so never changed; below the moon the other four were jumbled and in turmoil. Wood burned because its fire and air "tended" upward while its earth tended down, and wood floated because of that fire and air, while stones and ashes (concentrated earth) sank.
Trouble came from astronomy, the most advanced empirical science for practical reasons: the calendar fixed religious holidays and sailors venturing into unknown waters needed better tables. Squaring observations with geocentrism, especially planetary positions, became so complicated that Spain's King Alfonso X reportedly exclaimed in the thirteenth century that God would have received "good advice" had He consulted him.
A doubtful anecdote, flagged by Haugeland himself. In a note he reports that historian B. R. Goldstein considers the Alfonso story most likely political slander spread by the king's opponents, exploiting a prophecy that blasphemy would ruin him and his known taste for astrology; the earliest surviving version is in a chronicle of 1344 found by Jerry Craddock. Haugeland's point needs only that the story was told and believed.
Nicolaus Copernicus (1473-1543) published On the Revolution of the Spheres in the last year of his life. Its heliocentric theory was "the most disorienting scientific innovation of all time", built on two claims:
- The daily motions of the heavens are an illusion, produced by the Earth rotating on its axis.
- The Sun's annual circuit through the Zodiac, and some of the planets' stranger wanderings, are equally illusory, produced by the Earth orbiting the Sun.
Earth became just another planet, between Venus and Mars. The consequences spread to all of physics: natural places were dislocated (which way is up, and where are Heaven and Hell?), and motion theory faced hard questions (why don't loose objects fly off the spinning Earth, and why doesn't a dropped stone land west of where it was released?). Solving them took the laws of inertia, force and gravitation.
The first wedge between mind and world
The modern mind was "invented" in this ferment, and its first impulse was the Copernican distinction between appearance and reality. Plato had already said things are never really what they seem: perceived objects are shadows on a wall cast by the eternal Forms. Copernicus needed something more drastic. The sunrise is no pale copy of the Earth's rotation; it is totally different. Once appearances need not even resemble reality, science can discover almost anything, and at the same stroke resemblance is undermined as the link between mind and world, driving the wedge that eventually transformed our understanding of thinking.
Galileo and the New Science
Galileo Galilei (1564-1642) brought the telescope into astronomy, discovering Jupiter's moons, the Moon's mountains and the phases of Venus, all ammunition for Copernicanism; the Inquisition sentenced him to house arrest for his last eleven years. For the history of mind, though, what matters is his application of mathematics to motion. Nature, he wrote, is a great book written "in the language of mathematics" whose characters are triangles, circles and other figures; without them one "wanders in a dark labyrinth". For Galileo mathematics meant geometry and arithmetic, used where we would write algebraic equations.
Lines that are not lines
The theorem itself (the mean speed theorem) matters less than how geometry is used. Traditionally geometry studied figures in space; Galileo let lines represent times, speeds or any physical variable.
- Time as a line. Point A is the "starting point" in time, not space; points further down AB are later instants, not later positions.
- Speed as segments. Equal segments across to GF represent constant speed; segments lengthening from AB to AE represent uniform acceleration. GF and AE represent nothing themselves; they define speed as a function of time.
- Distance as area. Distances become "aggregate" areas (a note observes this anticipates the integral calculus of Newton and Leibniz some fifty years later), so the proof collapses into the triviality that two figures have equal area.
- The payoff. With the hypothesis that falling bodies accelerate uniformly, Galileo proved the "times-squared" law (distance from rest ∝ time²), and by treating a projectile as uniform horizontal motion plus vertical free fall he showed its path is a parabola.
Galileo's achievement was the abstract representation in which such proofs could be given. Once motion had a geometric notation, Euclid's entire deductive system could be lifted off shapes and applied to motion. A notation's power lies in the reasoning it licenses, not in what it looks like.
Secondary qualities
Galileo also drew a famous metaphysical conclusion. To excite tastes, odours and sounds, external bodies need only size, shape and fast or slow motions of countless tiny particles; take away ears, tongues and noses and shapes, numbers and motions remain but not odours or sounds, which are "nothing but names" outside the living animal, just as tickling is nothing apart from the armpit. He put colours and heat in the same class. The idea goes back to the atomist Democritus (fifth century BC), but Galileo made it credible by actually deciphering nature's mathematical characters. John Locke (1632-1704) later named the two groups primary and secondary qualities (probably borrowing the terms from the chemist Robert Boyle). The distinction drives the Copernican wedge deeper, separating how things seem from how they are.
Hobbes, the Grandfather of AI
"By RATIOCINATION, I mean computation," proclaimed Thomas Hobbes (1588-1679) in the 1650s, "prophetically launching Artificial Intelligence". The slogan packs two ideas:
- Thinking is mental discourse. Thought consists of symbolic operations like talking aloud or calculating on paper, but conducted internally in special brain tokens Hobbes called "phantasms" or thought "parcels".
- Clear thinking follows methodical rules. At its most rational, thinking is like an accountant applying exact rules of calculation, a "mechanical" process like a mental abacus; when the rules are bent, the person is simply being irrational.
In Leviathan Hobbes says that reasoning is nothing but conceiving a sum total from the addition of parcels or a remainder from subtraction, and that this holds for geometers' lines, angles, "times, degrees of swiftness, force, power", and for logicians adding names into affirmations, affirmations into syllogisms and syllogisms into demonstrations. Haugeland forgives the overstretched "addition" metaphor and notes the giveaway: times and swiftness are Galileo's. Hobbes admired Galileo, travelled to Italy to meet him in 1634, and extrapolated from "the study of motion is geometry" to "thinking is computation". He tried to do for politics what Galileo had done for physics, earning the title "father of modern political science".
Hobbes went one step past Galileo: if reality is only particles in motion, so are the mind and its contents. "Motion produceth nothing but motion." Blue or tickling sensations from a feather are patterns of tiny motions in sense organs and brain, so when Hobbes speaks of putting thought parcels together he means real physical manipulation of tiny physical symbols.
- Hobbes: the "grandfather" of AI, who first said thinking is computation.
- Turing: the father, who planted the seed.
- McCarthy: the godfather, who gave the field a name and a place in society.
- Newell, Shaw and Simon: the ones who actually brought it into the world with heuristic search programs (Haugeland's genealogy, given in a note).
The mystery of original meaning
Hobbes's program has a fundamental flaw: it cannot tell the difference between minds and books. The question is how thought parcels can mean anything. The analogy with spoken and written symbols is no help, because word meanings are derivative: "horse" means a kind of animal only because English speakers mean that by it. Hobbes himself agreed that the word "stone" signs a stone only because the hearer gathers that the speaker is thinking of stone. Explaining thoughts the same way would be circular.
Haugeland calls this the mystery of original meaning: some meanings derive from others, but not all can, so thoughts must have original meaning while words have only derivative meaning. Whatever "further difference" gives thoughts their originality is the crux of the matter, and Hobbes had nothing to offer.
The resemblance theory had an answer (thoughts mean because they picture), but Hobbes could not use it. Modelling thought on discourse and computation gives thoughts the structure of sentences or formulae built from distinct, arbitrary symbols. Images lack that structure: a picture of a fat man running is not composed of three symbols for "fat", "man" and "running", so rules for proofs and derivations do not apply to images. By proposing computation, Hobbes forfeited resemblance, and was left with no account of original meaning.
The proposal: my experience is of this apple because this apple causes it, memories inherit the causal link, and perhaps the concept 'apple' gets its meaning from the apples used to teach it. Neither derivative nor resemblance-based, it still faces four questions.
- Which causes? Overindulgence causes splitting headaches, yet headaches do not mean overindulgence; my apple experience is also caused by photons, my opening my eyes and the skin's microtexture. Why do only some causes generate meanings?
- Which meanings? Looking at one apple I can think of an apple, a fruit, a gift from a student, lunch, its redness, its distance, all different contents with the same causal connection.
- Informed meaning. Mechanic and client both hear an engine ping, but it means more to the mechanic, in experience and in concept.
- Noncausal objects. I can think of the number eleven, the future, unrealised possibilities and the species raccoon, none of which has caused anything.
Haugeland allows that causal factors may form part of a richer theory, and notes that much current work goes that way using ideas unavailable to Hobbes.
Causal and teleological theories, and the LLM version of the mystery. The "current work" Haugeland mentions became a major industry: Fred Dretske's information-based account (Knowledge and the Flow of Information, 1981), Jerry Fodor's asymmetric-dependence theory (late 1980s) and Ruth Millikan's teleosemantics (1984) all tried to answer "which causes?" and the problem of misrepresentation; none won consensus. The mystery of original meaning returned forcefully with large language models, which are trained on human text whose meaning is plainly derivative. A widely cited 2020 paper by Emily Bender and Alexander Koller argued that a system trained only on linguistic form cannot learn meaning; critics reply that multimodal training, tool use and interaction with the world supply some grounding, or that meaning partly consists in patterns of use. Haugeland's Hobbes problem (telling a mind from a very good book) is now an engineering-adjacent question.
Descartes
Copernicus and Galileo were great physicists but modest philosophers; Hobbes was a great philosopher, "an inconsequential physicist and a truly abysmal mathematician". René Descartes (1596-1650) was a world-shaker at everything: "father of modern philosophy", arguably father of modern mathematics, and for a generation the foundation of physics until Newton eclipsed him. All his work converges on the theory of mind.
Analytic geometry and its interpretation
Descartes made two mathematical innovations. The first was analytic geometry: points, lines and relations represented by numbers and algebraic equations on what we call Cartesian coordinates. Coordinate ideas were old (astronomers tracked stars that way); his contribution was a systematic method of solving geometric problems algebraically, plus most of the algebraic notation still in use. La Géométrie opens by claiming any geometry problem reduces to knowing the lengths of a few straight lines.
Geometers such as Galileo and Kepler expressed quantities as tedious proportions ("line A is to B as C is to D"). Descartes treated proportions and arithmetic operations as forms of one more general relational structure, and so could multiply, divide and take roots of line segments and get line segments back. Geometrically this is bizarre (a product of lengths should be an area). His trick was to fix a unit length and define the operations as proportions:
- Squares. If L is to R as R is to 1, then L = R².
- Products. If 1 is to A as B is to C, then A × B = C.
Every geometric proportion can therefore be rewritten as an equation and solved algebraically. Then the leap: if physical laws can be geometric and geometry can be algebraic, physics can be written in algebra. We forget how big a step that was, and that rigorous quantitative science existed before algebra (Galileo and Kepler did it with geometry).
The second innovation was redefining mathematics. In the Discourse on Method Descartes says the mathematical sciences treat different objects but consider only the relations or proportions in them, so he studied proportions in general, free of any subject matter, the better to apply them to whatever objects they fit. Galileo abstracted Euclid from space to motion; Descartes drew the radical conclusion that geometry, algebra and physics are all "applied math", and mathematics proper concerns only abstract relationships.
A new theory of meaning
These discoveries were equally revolutionary for the philosophy of mind, because their core is a new view of how symbols relate to what they symbolise. Descartes extended it to thoughts, treating them as symbolic representations analogous to mathematical notation.
- The negative half: rending symbol and symbolised apart. Algebra does not intrinsically represent numbers, and Euclidean notation does not intrinsically represent figures; each is equally adequate for numerical, spatial or kinematic relations. Extend this to thoughts and the divorce of thought from thing is final: my thoughts of numbers are no more tied to numbers than algebraic notation is, and the same thoughts could represent spatial or kinematic facts, or nothing at all. "This disconcerting picture is the essence of the modern mind; we owe it all to Descartes."
- The positive half: reuniting them. Read between the lines and Descartes splits this into two questions: (1) what makes a notation suitable for representing a subject matter, and (2) what makes a suitable notation actually represent it.
To question 2, which is the mystery of original meaning again (external symbols only borrow meaning from thoughts), Descartes gave an "elaborate and sadly influential" answer. In barest outline: he can prove from his own mind that a nice God exists; a nice God would not leave conscientious creatures radically misguided; so conscientious thoughts represent reality. Its main effect was to send others looking for better answers.
Question 1 Descartes barely answered, so Haugeland supplies an answer by "creative anachronism". Inventing symbols for items is trivial; what earned Galileo and Descartes their reputations was showing that represented in specific ways, problems could be solved.
- The criterion. A notation is suitable for a subject matter when it can be used to solve problems about it. Galileo's derivations showed Euclid suits kinematics; Descartes did the same for algebra and geometry and saw that many notations may suit many subject matters.
- Corollary 1: integration. Scattered successes would look like flukes or hoaxes, so notation plus method must form an integrated system used systematically and reliably over a well-defined area.
- Corollary 2: rules. Solving problems needs allowable steps from the representation of a problem to the representation of its solution (derivations, proofs), so the system must include rules saying which steps are allowed.
Descartes's machine test
Rules and steps "bubble beneath" Descartes's famous distinction between people and unreasoning machines (1637). We can imagine a machine that utters words, even asks what you want when touched in one place and cries out that you are hurting it when touched in another, but no machine could arrange its words in various ways to respond to the sense of whatever is said, "as even the dullest people can do". Haugeland marvels that Descartes imagined the loudspeaker and "the talking Toyota" and anticipated Turing's test. Responding to the sense of an utterance means saying something reasonable, and reasonableness is fixed by rules for manipulating thought symbols. So Descartes's claim is that machines cannot think because they cannot manipulate symbols rationally, which hits AI "where it lives". (A note adds that Descartes put animals among the unreasoning machines, which spared him from granting them souls.)
- The classic doubt. Senses deceive (illusions, hallucinations, dreams) and reasoning errs, so how can we know we are not being fooled right now, and hence anything at all?
- Descartes's escalation. Imagine a divinely powerful, malicious "evil demon" who plants sense impressions and concepts bearing no relation to the world yet fitting together perfectly, so that one's whole life is an orchestrated hallucination.
- Why he could imagine it. The sunrise illusion still depends on the real Earth turning, and Galileo's tickle on the feather's motions; the demon cuts all contact with reality. Descartes could conceive this only because he saw thoughts as symbols in a notation that could represent one subject, another, or none, with the system itself unchanged.
- Haugeland's verdict. Epistemological skepticism has been largely a digression and a waste of effort, and it is tangential to AI.
The Paradox of Mechanical Reason
Descartes was a dualist: minds and the physical universe are two different substances. Thoughts are "in" minds in a non-spatial sense; bodies are in space with definite size, shape and location. So no mind contains a physical object and no thought is in the physical universe. Dualism has strong common-sense appeal and would rule out AI at a stroke. When Frank remembers Frankfurt or hankers for a frankfurter, is the thought an inch wide, doughnut-shaped? How far is his recollection from his left earlobe? The questions are perverse, which favours dualism.
Dualism's unresolved difficulty is interaction, the mind-body problem. Physics explains every movement of every particle by forces fixed by physical properties (mass, distance, charge, "queer quark quirks"). Thoughts with no size or location have none of these, so they can exert no force, and admitting interaction would cost modern physics. Yet denying it is awkward: my decision to raise my hand seems to cause the hand to rise, and light entering my eyes seems to cause my experiences.
| Dualist desperado | Claim | Haugeland's jab |
|---|---|---|
| Interactionism | Mind and body do interact, via subtle vapours in the pineal organ (Descartes himself) | Sensible, except that it contradicts everything else Descartes believed |
| Parallelism | Mind and matter are two perfect clocks synchronised by God at creation | A crushed thumb and clouded judgement are a coincidence, like noon whistle and church bells |
| Occasionalism | On each occasion God intervenes to produce the matching mental or bodily event | God creates my pain and then forms on my lips the curses I intend |
| Epiphenomenalism | Matter gives off mind as a by-product, like a machine's ticking; mind affects nothing | We watch the world go by; our sense of acting is "a cruel hoax" |
The alternatives are monisms. Idealism (minds and ideas are the only reality) was the nineteenth century's favourite and has since fallen on hard times for reasons irrelevant to AI. Materialism (only matter is real) dominates now, and its crux is what to say about thought. Haugeland calls that crux the paradox of mechanical reason, and its resolution is both the philosophical foundation of the AI boom and the most attractive alternative to "hopeless dualism and zany idealism (not to mention vulgar behaviorism)".
- Premise. On the computational model, reasoning is the manipulation of meaningful symbols according to rational rules in an integrated system, so there must be a manipulator.
- Horn A (the manipulator attends to meanings). Then it cannot be entirely mechanical, because meanings exert no physical forces.
- Horn B (it ignores meanings). Then its manipulations cannot be reasoning, because what is reasonable depends on what the symbols mean.
- Conclusion. If a process is mechanical, it cannot reason; if it reasons, it cannot be mechanical.
Horn A and the homunculus
Taking meanings into account re-enacts the dualist interaction problem inside materialism. Materialists say thoughts are material symbols, so they can interact with matter, but the meanings of those symbols are not material, and they must still steer the mechanism. Suppose that could be fixed. A second problem remains: who is the manipulator? Imagine it at work in Zelda's mind: it reads her mental symbols, works out what they mean, looks up rules of reason, decides which apply and writes new symbols. It is pretty smart. It cannot be Zelda herself, because Zelda's thoughts and understandings are the symbols being manipulated; if the manipulating needs thoughts and understandings, they must be someone else's, or the explanation is circular.
To explain thinking, the meaning-respecting horn must posit an inner manipulator that thinks, understands and decides on its own: the homunculus (Latin for "little fellow"), politely renamed the "faculty of the will" or "transcendental ego". Then what explains how the homunculus thinks? A still smaller homunculus?
Hume, the Mental Mechanic
David Hume (1711-1776), Scotland's most celebrated philosopher, was the first to spell out the mechanical conception of thinking consistently. His Treatise of Human Nature, written in his twenties, is subtitled "An Attempt to introduce the experimental Method of Reasoning into Moral Subjects", meaning the methods of natural science applied to the "science of man", starting with psychology. He wanted to discover "the secret springs and principles" of the mind as Isaac Newton (1642-1727) had found the laws governing the planets: to be the Newton of the mind.
The centrepiece is the association of ideas, taken nearly intact from Locke, who took much of it from Hobbes. Locke's mind starts as a tabula rasa on which experience writes sense impressions, which association combines into ever more complex knowledge; his aim was to legitimate Newton's empirical method against the Cartesians, using physics as an acid test for theories of mind. Hume imitated Newton directly: impressions and ideas were the "corpuscles" of a mental mechanics, and association was the umbrella for all mental forces, "a kind of ATTRACTION" with effects as extraordinary in the mental world as in the natural, plainly modelled on gravity.
Hobbes. Thoughts are physical symbols like numerals or words, manipulated by rules of calculation or discourse. Rule-following seems to need an intelligent rule-follower, so Hobbes is exposed to the homunculus.
Hume. The analogy with physics is at the level of explanation: ideas are like particles in that natural forces under natural laws move them (he was largely indifferent to materialism versus dualism). Ideas obey association as planets obey gravity: they just do, so there is no hidden intelligence and, like Newton, Hume can say "I frame no hypotheses".
Escaping the homunculus cost Hume everything mental. What makes his ideas ideas, and their interactions thinking? He eliminated meaning and rationality so thoroughly from the basis of his account that it might as well be a strange new physics, and he never explains how meaning and reason get back in. He is impaled on Horn B: meanings do not matter, so the processes are not rational. Historically, transcendental idealism (Kant, Hegel and their legions) returned to Horn A and gave up on matter; Haugeland skips that episode as a digression from AI's pedigree and resumes with "confident and all-conquering twentieth-century materialism".
- The target. Hume attacks the "double existence" theory: fleeting perceptions in us, and durable external objects that cause and resemble them.
- The argument. We only ever work with perceptions; we never experience objects independently of them, and the existence of perceptions never logically implies anything else, so "external objects" are figments of imagination.
- The consequence. On Hume's theory mental representation of nonmental objects is inconceivable, so representational content has no place in it.
- The contrast. Descartes's skeptic asks "How can I know what I know?"; Hume's asks the deeper "How can I even mean what I mean?"
| Thinker | Dates | Contribution to the modern mind | Problem left open |
|---|---|---|---|
| Copernicus | 1473-1543 | Appearances need not resemble reality | Physics of motion dislocated |
| Galileo | 1564-1642 | Notation can represent any variable it reasons about; secondary qualities | Says little about mind |
| Hobbes | 1588-1679 | Thinking is computation on physical parcels | Original meaning; minds vs books |
| Descartes | 1596-1650 | Notation has no intrinsic subject; suitability = problem-solving with rules | Mind-body interaction; meaning via a good God |
| Hume | 1711-1776 | Mental mechanics by association, no homunculus | Nothing mental or rational remains |
What the saga leaves out. Haugeland admits the streamlining. Two consequences for the reader: the story runs straight from Hobbes to symbolic AI, so it gives no place to the associationist line through Hume, which was in fact the ancestor of connectionism and today's neural networks (learning by association of co-occurring patterns); and it leaves Leibniz out of the philosophical saga (he surfaces later only as a builder of calculating machines), although his dream of a universal calculus of reasoning belongs squarely in a pedigree of "thinking as computation". Haugeland's dismissal of skepticism as wasted effort is also a personal verdict that many epistemologists would reject.
The book that thinks. Imagine a library so complete that for any question you type, a clerk with no understanding finds a page whose text answers it sensibly, in context, every time. Nothing in the building means anything originally: every sentence borrows meaning from its human authors. Is the system a mind or a very large book? Hobbes's theory cannot say, and that is exactly Haugeland's charge. The experiment isolates the question every later chapter must face: what extra ingredient, if any, turns derivative meaning into original meaning?
Automatic Formal Systems
To see how the paradox of mechanical reason might be resolved, we need to know what a computer really is, and Haugeland's definition is exact: a computer is an interpreted automatic formal system. This chapter builds the first part, automatic formal systems, from the ground up: tokens, positions and legal moves; digitalness; medium independence and formal equivalence; finite playability and algorithms; complex tokens; and finally automation, with the Automation Principle and the trick that lets fallible heuristics run on infallible algorithms. Meaning is held back for Chapter 3.
Pocket calculators make the paradox feel quaint ("obviously there can be reasoning machines"), but in philosophy nothing is so treacherous as the obvious. Even if the paradox is now resolved, we must see which new ideas broke through where Hobbes, Descartes, Hume, introspective psychologists, psychoanalysts and behaviourists failed.
Formal Games
A formal system is like a game in which tokens are manipulated according to rules, to see what configurations can be obtained. Chess, checkers, Chinese checkers, go and tic-tac-toe simply are formal systems; marbles, tiddlywinks, billiards and baseball are not. All formal games share three essential features: they are token-manipulation games, they are digital, and they are finitely playable.
Tokens, positions and moves
Tokens are the pieces: chessmen, go stones, but equally O's and X's in pencil or the switch settings and light patterns of electronic games. Manipulating tokens means one or more of four things:
- Relocating them on a board or field.
- Altering them, or replacing them with different ones.
- Adding new ones to the position.
- Taking some away.
Chess pieces are mostly moved, sometimes removed (captured) and occasionally altered (promoted), but never added; in go a stone is added each play and some are occasionally removed, but stones are never moved or altered. Any manipulation can be called a move.
- Tokens. Specify what the tokens are.
- Starting position. Specify the starting position or positions (chess has one; go has several, depending on the players' relative strength; some systems allow any number).
- Rules. Specify which moves would be legal in any given position.
- Position. A position is an arrangement of tokens at a given moment; play proceeds from a starting position by legal moves only.
Self-containedness. What the rules permit depends on the current position and nothing else: a move legal once in a position is legal whenever that position recurs. The outside world is strictly irrelevant: a chess game is unaffected if the set is stolen, the building is on fire or the fate of nations hangs on it. (A note adds that formal systems are therefore ahistorical, so fully formal chess needs a few side tokens recording whether king and rook have moved, for castling.)
The decisive consequence of self-containedness is the irrelevance of meaning. If the string "The cat is on the mat" is part of a game position, nothing about any cat or mat makes a difference to which moves are legal. Meaning is not a formal property, because meanings relate to the outside world.
Games usually designate goal or "winning" positions, and players compete to reach them, but competition and goals are not essential; they are there for fun, and many important formal systems (computers above all) matter for other reasons.
Peg solitaire
The standard non-competitive example is peg solitaire on a cross-shaped board of 33 holes, starting with 32 pegs and only the centre empty. There is one kind of legal move: if two pegs are adjacent in a row or column with an empty hole next to one of them in line, jump one peg over the other into the hole and remove the jumped peg. The goal is to leave one peg, in the centre. Haugeland insists this is not an analogy for a formal system; it is a formal system.
Token types
Most interesting systems have several token types (chess has six per side: king, queen, rook, bishop, knight, pawn). Tokens are freely interchangeable if and only if they are of the same type; swapping two white pawns changes nothing, swapping a pawn and a rook changes the position. "Altering" a token means changing its type.
Ultimately, the rules determine the types: two tokens are of the same type if interchanging them would never make any difference to which moves are legal in any position. In a fancy set where every pawn figurine is unique, they are all pawns because each contributes to every position in exactly the way pawns do.
Digital Systems
"Digital" suggests computers and numeric readouts, but it is a fundamental metaphysical category. Every formal system is digital, but not everything digital is a formal system.
| Digital | Not digital |
|---|---|
| The alphabet | Photographs |
| Modern currency | Gold bullion |
| Playing cards | Pick-up sticks |
| Rotary function switch on a stereo | Volume control |
| Poker chips | Coloured sand |
| Chess, solitaire | Billiards |
The full definition: a digital system is a set of positive and reliable techniques for producing and reidentifying tokens, or configurations of tokens, from some prespecified set of types. Simplified: a digital system is a set of positive write/read techniques, with two warnings. "Writing" is any manipulation that changes the formal position (putting a peg in the centre hole writes a token of the type peg in centre hole). "Reading" implies no understanding or even recognition, only differentiation by type and position: an elevator "reads" button presses, as shown by stopping at the right floors.
Positive versus reliable
- Positive. A positive technique can succeed absolutely, totally and without qualification, as opposed to "almost exactly" or "close enough for government work". Sawing a board to between 5 ft 11 in and 6 ft 1 in is positive; sawing it to exactly six feet is not, since it can succeed only approximately.
- Reliable. Reliability is the likelihood of success, a separate question. Within an inch of six feet is positive and reliable; within a sixteenth is positive but reliable only for a skilled sawyer; within a millionth of an inch is neither, because wood cannot be cut that precisely.
- Examples. Shooting a basketball is positive (it can go through absolutely) and reliable for good players; counting pencils is positive; closing a switch to start a furnace is positive and thermostats do it reliably. But no technique positively holds a room at exactly 68 degrees, weighs the pencils exactly, or puts a ball exactly through the centre of the hoop.
- Success criterion. A write/read cycle succeeds if the type read is the type that was specified to the writer. A digital system is a set of write/read techniques positive and reliable relative to that standard. (A note adds that write tolerances are set by read precision and vice versa: it is the cycle that must be positive and reliable.)
A multiposition switch is a clean digital device: positions are separated by click stops and it is unambiguous which circuit conducts.
Rembrandt's portraits. Even with the finest care the paintings are slowly deteriorating and are no longer as they were. Paint colours and textures cannot be exactly duplicated even in one medium.
Shakespeare's sonnets. Apart from a few lost or miscopied, we probably have most of them exactly as written, absolutely without flaw, because the alphabet is digital. Two copies in different handwriting or fonts are equally perfect if each character is clearly recognisable.
Digital systems achieve perfection despite the world by allowing a margin for error within which all performances are equivalent and success is total: "a basket is a basket is a basket; and a miss isn't."
Why digital matters: poker chips and sand
Two ways of keeping poker money use blue, red and white for 100, 10 and 1. One uses plastic chips; the other uses tablespoons of coloured sand. Sand allows fractional bets, but chips are digital: counting is a positive technique, so every bet is exact.
- Assumed sand error: about ±2% on any measured volume.
- Wager: 914 units, which with chips is exactly nine blue, one red, four white.
- Error on the blue sand alone: 2% of 900 = 18 units, which is more than all the red and white sand combined (14 units).
- Consequence: imprecision in the large denomination swamps the small ones and makes them superfluous; a scratched blue chip is still worth exactly 100 white chips.
The principle generalises: in any complicated system where tiny deviations in one part can make all the difference to other parts, something must stop the influential components from overwhelming the sensitive ones, and digital design is one of the most powerful means known. That is why formal systems can become extraordinarily complicated without getting out of control.
Solitaire is digital because a peg leaning slightly is still absolutely in that hole and no other, "as any child can see". Chess adds piece types, but discriminating them is trivial. Billiards is not digital: jostle a tournament table and the position can never be perfectly reconstructed, even with photographs, while chess players with good memories can restore a disturbed game exactly.
Medium Independence and Formal Equivalence
Digitalness supports another property: medium independence, meaning the same formal system can be materialised in any number of media with no formally significant difference. Intelligence must share this property, "if there are ever to be smart robots".
- Solitaire in any medium. Pegs may be wood or plastic, red or green, any height, or replaced by marbles in craters, chalked X's, or onion rings hanging on nails; change the starting position or jump rule, though, and it is a different game.
- Chess in any medium. Each piece need only be positively identifiable by type, side and square. Texas millionaires could play from facing penthouses with 32 radio-controlled helicopters on 64 rooftops, or with marked window shades in an eight-storey hotel with eight rooms per floor.
- Non-formal games are tied to their medium. Doubling or halving a football changes passing and kicking strategy; magnify it twenty times and no football is possible. Billiards cannot be played with pegs or onion rings.
- Electronic toys. Pocket hockey resembles hockey as stick figures resemble people, but electronic chess is chess, perhaps not grandmaster quality, but the real thing: a quarterback cannot pick up a blip, yet luminous displays are full-fledged chess pieces that Russian champions can attack.
Chess seems less "physical" than football not because it is intellectual (sports demand strategy, and chess pieces are physical objects) but because nothing in it depends on specific features of its medium. Digitalness makes medium independence feasible: poems can be chiselled in stone, embossed in Braille, stored in computer code or traced in sand, unaffected so long as letters and punctuation keep their order. Medium independence does not make media arbitrary, though: puffs of smoke are too unmanoeuvrable and fleeting, and a welded steel chess set cannot be moved. Tokens must be manipulable enough to write and durable enough to read, so not everything can be digital.
Formal equivalence: the chip game
Haugeland defines an unfamiliar game. Thirty-three plastic chips each carry an early letter (A-G) and a late letter (T-Z). At the start all are in a white basket except DW, alone in a black basket. A move exchanges two chips in the white basket for one in the black, provided either all three share the early letter with sequential late letters or share the late letter with sequential early letters, and the middle chip of the sequence is not the one coming from black. So AX, BX ⇔ CX and AX, AW ⇔ AV are legal; ET, EV ⇔ EZ (not sequential), AW, BX ⇔ CY (no shared letter) and AX, CX ⇔ BX (middle letter from the black basket) are not. The goal is to end with every chip but DW in the black basket.
Laid out as above, each chip is a hole on the solitaire board; transferring a chip to black is removing a peg, transferring one back is filling a hole, and the exchange rule matches the jump rule exactly. Yet the tokens do not correspond: 32 identical pegs versus 33 distinct chips, 33 places versus 2. Token distinctions are traded for place distinctions, giving the same number of possibilities. The correspondence is complete and exact but only at a more abstract level, relating whole positions rather than individual tokens.
Two formal systems are formally equivalent just in case:
- For each distinct position in one there is exactly one corresponding position in the other.
- Whenever a move would be legal in one, the corresponding move (from corresponding position to corresponding position) would be legal in the other.
- All the starting positions correspond.
Then "if you can get there from here in one, you can get there from here in the other"; for games, the goal positions should also correspond. Medium independence is the special case where tokens correspond one to one, as with helicopter chess. Chess by mail, with formulae like P-K4 on postcards, is equivalent to both.
The payoff for AI, stated crudely: brain cells and electronic circuits are different media, but at some level of abstraction they may be media for equivalent formal systems. If so, a computer mind might be as genuine a mind as computer chess is genuine chess, only in a different medium.
Multiple realisability, then and now. Haugeland's medium independence is the computer-science version of what philosophers call multiple realisability, the backbone of functionalism. It remains contested. Biological naturalists in the tradition of John Searle deny that the right formal organisation in any medium suffices for mind, and integrated information theory denies that it suffices for consciousness (on IIT the physical causal structure matters, so a simulation on conventional hardware would not be conscious), while computational functionalists accept medium independence. Modern machine learning added a twist Haugeland could not foresee: a trained network's behaviour survives quantisation of its weights to 8 or even 4 bits, a striking real-world case of a system that keeps working "within tolerances", yet its internal states are real-valued activations, not discrete meaningful tokens, so whether a neural network is a "formal system" in his sense depends on the level of description one picks.
Finite Playability and Algorithms
Formal games must be playable by finite players: no infinite or magical powers, but also no gratuitous parochialism (players are not limited to human or present-day capacities). Playing means following the rules, which requires being able, in any position, to:
- Tell, for any proposed move, whether it is legal.
- Produce at least one legal move, or show there is none.
Primitive operations
Some operations are manifestly finite because trivial: an electric typewriter types an A when a switch closes; vending machines sort coins; moving a piece to an indicated square is easy. A finite player has a finite repertoire of primitive operations it can perform positively and reliably any finite number of times. Chess, though, needs non-primitive tests (is this legal for the piece, does it expose the king, is it blocked?), and these are built as ordered combinations of primitives specified by a rule or recipe. That lands us back at rule following: "it involves following rules. Wonderful!" The way out is a special class of rules needing only primitive rule-following abilities.
An algorithm is an infallible, step-by-step recipe for obtaining a prespecified result.
- Infallible. It is guaranteed to succeed positively in a finite number of steps, if each step is done correctly, though there need be no prior limit on how many.
- Step-by-step (1). It prescribes one step at a time.
- Step-by-step (2). After each step the next is fully determined: no options or uncertainties.
- Step-by-step (3). After each step the next is obvious: no insight or ingenuity required.
Algorithms are mindless "turn the crank" routines that always work sooner or later: trying every key on a ring, alphabetising lists, mating a lone king with king and rook, transposing songs, multiplying integers in Arabic numerals. They are specified as input/output relations.
From straight schedules to branching
The simplest recipe is a straight schedule (do A, then B, ... then Z). Following one needs only two primitive rule-following abilities: (1) obeying primitive instructions (think of each typewriter key as a primitive instruction) and (2) moving to the next instruction in a finite list (a typewriter fed a paper tape of keystrokes). Telling what to do is distinct from doing it, and a player needs both. Straight schedules are inflexible: they cannot cope with keyrings of any size (different inputs) and cannot stop when the right key is found (intermediate results).
The capitalised conditional branch directives ("DID THE LOCK OPEN? IF SO, GO TO LINE 4") are signposts, not operations; they need two new abilities, answering yes/no questions about the previous step and branching accordingly. The loop back to line 2 handles any number of keys. The red ribbon matters more than it seems: since the loop must be able to run without limit, the recipe also needs a successful stopping condition when no key works, or it is not an algorithm. A vending machine shows finite players can answer primitive yes/no questions and follow alternative branches, so finite branched schedules are followable. Haugeland names the result: a primitive algorithm is a branched-schedule algorithm with primitive branching conditions and primitive instructions, and executing primitive algorithms is within the compass of finite players.
- Brute force. To find the smallest positive integer passing a test with no analytic solution, try 1, then 2, and so on; if some number passes, this is an algorithm, but if none does it never terminates.
- Termination is the point. In a formal game, legality tests must always terminate with yes or no; that is what finite playability means.
- No oracles. A crystal ball answering otherwise undecidable cases would not make a game finitely playable, because if magic were allowed, the finitude restriction would be vacuous.
- Why mechanisms. Magic is unintelligible, but not everything unintelligible is magic: pioneers of formal mathematics could not be sure whether mathematical "insight" and "intuition" were magical. The only certain way to abjure magic is to insist on transparently intelligible procedures such as simple physical mechanisms, hence the value of typewriters and vending machines as examples, and why algorithms are always "mindless and mechanical".
Algorithms all the way down
Because executing a primitive algorithm is within finite competence, a whole primitive algorithm can serve as a single primitive step in a higher algorithm, and so on up. The keyring routine becomes step 2 of a recipe for trying every lock around a fortress (marked with a blue flag), and without the red ribbon that higher recipe could never get past the first lock. For chess, primitives (identifying pieces and squares) build algorithms for a piece's range, how much is blocked, which moves expose the king, then a legality test, then an algorithm that lists every legal move in any position. It makes no difference how many levels pile up, "as long as it's algorithms all the way down"; only the ground floor must be genuinely primitive, and with "these bricks, and algorithmic glue, there's no limit to how high we can build."
Complex Tokens
Some tokens are built from simpler tokens: sentences from words, words from letters, numerals from digits. What matters is whether internal composition systematically determines how a token can be used.
- Morse code is trivial. Letters are made of dots and dashes, but their structure has no significance beyond each being unique.
- Spelling is nearly trivial. It roughly guides pronunciation but tells you nothing about how a word is used to say things.
- Arabic numerals are not trivial. Faced with unseen numerals (783, 54,912, 99.44, 0.0043) you can use them correctly at once; faced with unseen words (dol, edh, mho, kylix, phiz, ootid, yagi) the spelling does not help at all.
Both examples cheat, since they trade on meaning, which formal systems exclude. The fix is to define a token's use as its contribution to the formal position (the difference it makes to which moves are legal). Haugeland then imagines learning multiplication as a pure game before knowing numerals are about numbers ("some children do learn arithmetic this way"): starting position 54912 × 783, then the stair-stepped partial products 164736, 439296, 384384, then the final line 42996096. The rules can be stated without mentioning numbers, so formal multiplication is a completely formal game.
Formal multiplication has infinitely many possible positions and moves, yet a finite player with a fixed repertoire can play it, because complex tokens are manipulated by recipes that go through their component digits one by one. Complex tokens plus complex recipes give infinite reach on finite competence, and the same holds for formalised mathematics, logic, linguistics and, if Hobbes is right, thought, whose formal structures would be enormous and elaborate, with composition critical to use.
- Simple and complex. Simple means having one part; complex means composed of several. Atoms are simple to chemistry but complex to atomic physics; nuclei are composite to nuclear physics; electrons have no known structure. Simplicity depends on point of view.
- Unbounded. Every integer is finite, but there is no upper bound on how big they get.
- Proof that there are infinitely many. Suppose there were only N finite integers; then they would be exactly 1 through N, but every integer has a larger successor, so no finite N can count them.
- Wider use. Formulae and proofs in logic must be finite but unbounded; an algorithm must terminate after finitely many steps with no limit on how many; so, in principle, with Turing-machine tapes and computer memories.
Automatic Systems
An automatic formal system is a formal system that plays by itself: a physical device in which (1) some parts or states are identified as tokens of a formal system and (2) in normal operation it automatically manipulates them according to that system's rules. Picture chess pieces hopping about legally by themselves, or a magical pencil writing correct mathematical derivations unguided; such systems, or equivalent ones, can now be built.
Playing a game requires players, who make moves, and a referee, who makes no moves but says whose turn it is, which tokens to work on, supplies the starting position, decides when the game is over and announces results. Friendly games blur the roles, but refereeing is always separate from playing. An automatic system is the manual game plus automated players plus an automated referee; electronic chess sets automate one player and the referee.
Black boxes and analysis
What players and referee are made of is irrelevant to what an automatic system is; they are black boxes, viewed only by their opaque surface. A concert sound engineer treats microphones, mixers and amplifiers as black boxes; the amplifier designer treats transistors and integrated circuits the same way. One engineer's component is another's whole system. Understanding how something does what it does by examining its parts and their organisation is analysis, and one kind explains a complex system by the interactions of its functional components.
In Haugeland's chess machine, an outer player is analysed into an inner automatic system. Its tokens are a copy of the outer position plus a move notation; its inner players are a move lister (generator) and a move chooser (evaluator); its inner referee shows the position to the lister, passes the list to the chooser and announces the chosen move as the outer player's move. The lister in turn has inner-inner players: a square selector, a piece selector and a legality tester. Analysis can descend boxes within boxes until components can be analysed no further.
The Automation Principle
The move lister is trivial in a deep sense, because there is an algorithm for listing all legal moves. Assign each primitive operation to its own player and give the referee all the primitive recipe-following abilities; the referee then follows the recipe, signalling players in order, and the algorithm executes automatically. Higher-level primitives are analysed into lower-level algorithms exactly as higher-level black boxes are analysed into lower-level systems, and the ground floor is identical, except that in an automatic system the primitives are carried out by primitive machines. That is where formal systems meet automation.
AUTOMATION PRINCIPLE: whenever the legal moves of a formal system are fully determined by algorithms, then that system can be automated.
- Physicalism (reductionism). Beyond materialism, the thesis that all sciences reduce to physics, the "unity of science".
- Nomological reduction. The standard case is thermodynamics: heat is energy in the chaotic motion of molecules, and thermodynamic laws are derivable from physical ones (nomos = law).
- Functional/systematic explanation. Mr Goodwrench explains a car engine not by physics but by the capacities and functions of its parts and how they fit together.
- Unexplained explainers. Laws used in an explanation are unexplained explainers; reduction explains one theory's unexplained explainers by another's, so physicalism holds there are none except physics's.
- Where automatic systems fit. Analysing them into ever smaller black boxes is reductionist in the functional/systematic sense, not nomologically; the link to physics comes at operations so primitive that mindless mechanisms can do them.
Choosing moves: heuristics
The move chooser is another story. Chess, like most interesting systems, is nondeterministic: most positions allow several legal moves. Algorithms are always deterministic. Two cheap answers (always take the first move on the list, or choose at random with electronic dice) yield legal but atrocious, chaotic chess that hardly deserves the name. The real problem is a chooser that chooses well, and there is no known feasible algorithm for telling in general which of two chess moves is better.
Champions do not play perfect chess either, so an infallible test is unnecessary; a fairly reliable one that usually eliminates the worst choices will do. Fallible but fairly reliable procedures are called heuristics: rules of thumb that are not algorithms, because algorithms must be infallible. Chess examples: capture pieces while losing none or only weaker ones, control the centre, activate pieces early, keep the king out of the way. Other things are not always equal (the "free" capture can be bait in a brilliant trap), so programs balance many heuristics and look ahead through sequences of moves.
| Algorithm | Heuristic | |
|---|---|---|
| Success | Infallible: guaranteed positive result in finite steps | Fallible but fairly reliable |
| Determinism | Fully determined, obvious next step | A rough, generic rule of thumb |
| Chess role | Move lister: generate all legal moves | Move chooser: estimate the best move |
| Automation | Directly, by the Automation Principle | Indirectly, via redescription |
The trick is redescription. Write a precise formula combining the heuristics; a routine that computes the formula's value is an algorithm if the goal is "calculate the estimate" but only a heuristic if the goal is "find the best move". Seen as an algorithm it can be automated; seen as an estimator, the same automated routine is a decent move chooser. A good but fallible move chooser can be automated by an algorithm after all, simply by describing the same routine from two points of view. Science does this constantly: physicians see patients as tissues and tissues as chemical stews; sonnets are preserved because we can see them as strings of digital characters. The most important redescription of all concerns meaning, the subject of the next chapter.
Chess engines since 1985. The generator-plus-evaluator architecture Haugeland sketches is exactly how classical engines worked, culminating in IBM's Deep Blue defeating Garry Kasparov in 1997 with deep search and a hand-tuned evaluation function. Since then the evaluator has been learned rather than written: AlphaZero (DeepMind, 2017) used a neural network trained by self-play with Monte Carlo tree search, and the leading open-source engine Stockfish adopted a neural evaluation (NNUE) in 2020. His claim that no feasible algorithm picks the better move in general still holds for the full game, but parts have been settled exactly: endgame tablebases give perfect play for all positions with up to seven pieces, and checkers was solved (a draw with perfect play) by Jonathan Schaeffer's team in 2007.
The algorithm-versus-heuristic redescription is the right mental model for LLM components in regulated workflows. The code that calls a model, parses its output and applies thresholds is a deterministic, auditable algorithm; what it computes is a fallible estimate of the right answer. Banking controls should be designed around the second description (error rates, escalation, human review), while change management and testing exploit the first (reproducible versions, fixed prompts, logged inputs and outputs). Haugeland's positive and reliable write/read idea is also why digital records, not model memories, must remain the system of record.
Limits of the formal-game picture. Defining formality by self-contained, digital token games makes the later move to minds look smoother than it is. Real cognition is embedded in a changing world (the very "outside" that formal systems exclude), and much of it seems graded and context-sensitive rather than digital. Haugeland himself turns to these worries in his final chapters; here the reader should note that medium independence is argued for games, not yet for minds, and the step from one to the other is the book's central hypothesis rather than a result.
The helicopter mind. Suppose every neuron in your brain were replaced, one at a time, by a radio-controlled helicopter hovering over a field, each changing altitude exactly when its neuron would have fired. If formal equivalence at the right level is all that matters, the helicopter swarm thinks your thoughts as surely as helicopter chess is chess. If you balk, ask whether your reluctance targets the medium (which Haugeland says is irrelevant) or the level (perhaps neurons are not the level at which thought is formal). The experiment shows that medium independence for minds depends entirely on finding the right level of description.
Semantics
Chapter 2 showed that formal systems are compatible with physical mechanism; this chapter asks what meaningfulness requires and puts the two together. Haugeland defines interpretation through one principle, coherence (an ordered text rendered so it makes reasonable sense), shows how interpreted formal systems let syntax guarantee semantics ("if you take care of the syntax, the semantics will take care of itself"), defines a computer as an interpreted automatic formal system, and names the research programme built on that idea: GOFAI, Good Old Fashioned Artificial Intelligence. He then claims to dissolve the paradox of mechanical reason by redescription, and finds the mystery of original meaning "battered but not down".
Meaningfulness
Modern philosophy posed, but did not answer, the question how meaning is possible in a physical, mechanical universe. It splits into two subproblems:
- What is compatible with physical mechanism?
- What is required for meaningfulness?
Chapter 2 made progress on the first by avoiding the second: formal systems are self-contained, so traditional mysteries about meaning cannot infect them, and even rule following can be fully mechanised. To understand AI, and minds, this must be joined to an answer to the second. Cognitive science follows Hobbes in assuming thinking is like talking, so linguistic meaning is a guide, with one difference kept in view: linguistic meaning is derivative, mental meaning must be original.
- Evidence. One thing helps discern another: "Clouds in the south mean a storm is coming." (We never say "the meaning of clouds is...")
- Significance. "Losing millions means little to him."
- Point or purpose. "Mystics ponder the meaning of life"; "What is the meaning of this?" asks Mother Superior, sternly.
- Intention to act. "Mrs O'Leary's cow didn't mean to burn down Chicago."
- Speaker's meaning. What someone intends to express: "She meant (by her grimace) that I should shut up."
- Semantic meaning. What an expression conveys in standard usage, what dictionaries explain and translations preserve: German "rot" means "red". This sixth sense is the chapter's topic.
Why thought might be like talk
The case is not that we sometimes "think in words" (rare, and the theory covers far more). There are deeper parallels.
- Speech acts express thoughts. Statements express beliefs, requests desires, apologies regrets, promises intentions, predictions expectations, with "thought" taken broadly to include hunches, hopes, horrors and worries, conscious or not. Roughly, we can say what we think, and when sincere we think what we say.
- Mode and content. "I assert / inquire / request / apologise that Fido eats his insects" share one propositional content in different modes; utterances in one mode can carry different contents (that Dinky drinks her milk, that emeralds are expensive). Thought has the same split: I can believe, desire or regret that Fido eats his insects. A speech act and the thought it expresses have the same content and corresponding modes: inquiries match wonderings, assertions beliefs, requests desires, apologies regrets. (A note credits this formulation to John Searle (1983), and adds that promises also create public obligations.)
- The abstract claim. Cognitive science does not say all thoughts are expressible in words. It says thought is like language in some abstract structural respect, compatible with thought being far richer or subtler.
- The meanings of simple symbols (e.g. words) are arbitrary: "squid" could as well have meant what "squalid" does; actual meanings are fixed by community usage.
- The meanings of complex symbols (e.g. sentences) are systematically determined by their composition: which simple symbols, and how they are put together (the "form" or "grammar"). This clause is compositionality.
The consequence is power. With a vocabulary of forty or fifty thousand words, a speaker understands literally trillions of distinct sentences, including ones never heard before (Haugeland's example has Mongolian lizards avoiding a dairy product, from which you can infer that some dairy products repel some Asian reptiles). Thought seems even more versatile, yet the brain is limited ("only ten billion neurons, more or less"), so how can a "small" system have so large a repertoire? The attractive answer is the linguistic one: thoughts are complex, built systematically from a modest kit of atomic constituents, and the versatility of thought comes from a combinatorial structure like the compositionality of symbolic systems. AI adopts this to the point of holding that thoughts are symbolic, without claiming we think in English: the inner system may differ vastly from ordinary languages while keeping compositionality and arbitrary basic meanings.
A dated figure. Haugeland's "ten billion neurons" was a common estimate in 1985. Counting by the isotropic fractionator method (Azevedo, Herculano-Houzel and colleagues, 2009) puts the human brain at roughly 86 billion neurons, with most of them in the cerebellum. The argument (finite brain, unbounded repertoire) is unaffected.
The theory: the meaning of a symbol is the very thing it symbolises, so 'Fido' means Fido, 'frog' means the class of frogs, and "Fido is a frog" means the fact that Fido belongs to it.
- Empty terms. "Santa is an elf" makes sense (children believe it) despite a worldwide dearth of Santas and elves; with no objects, the theory leaves it meaningless.
- Same reference, different sense. "Mile-high city" and "Colorado's capital" both pick out Denver but differ in meaning; "feathered" and "beaked" denote exactly the same animals (birds) but are not synonyms.
- Meaningful parts without objects. "Fido drips", "Does Fido drip?", "Fido does not drip", "Perhaps Fido could drip" are all about Fido and dripping; "does", "not", "perhaps" and word order have no objects yet change the meaning.
Interpretation
You are a crafty explorer who finds alien markings you suspect are symbols. To interpret is to make sense of; to interpret tokens as symbolic is to specify in a regular manner what they mean, normally in two parts: what the simple symbols mean and how complex meanings follow from composition. It is like producing a translation manual, a dictionary plus a grammar, into a system already understood, such as English.
The basic principle of all interpretation is coherence. It has two sides that are, at root, one principle: you must start with an ordered text, and you must render it so that it makes reasonable sense.
Ordered text
- Not accidental. In a far corner of the galaxy the stars spell, with revolting precision, a four-letter Anglo-Saxon vulgarism. Coincidence or a higher power who skywrites in English? If coincidence, it means nothing; it is not even a word. Order needs a source or reason: there is, or could be, "more where that came from". After the millionth monkey types a sonnet, you don't hold your breath for Hamlet. The actual text is a sample of the potential text, which must be consistently interpretable throughout.
- Enough structure. In a scrambled-alphabet cipher, "Abcd!" is hopeless: it could be "Fire!", "Whoa!", "Egad!", "Nuts!". A longer cryptogram, "Abcd aefg ahf jkkg.", is hard to satisfy: Haugeland found only three solutions, "Dumb dogs dig wells." "Cold cats cut riffs." "Rife rats rot hulls." He defies anyone to scramble a full page and find another sensible solution.
- Structure relative to the scheme. Sneaky Nikita rotates twenty scrambled alphabets, so a twenty-character message offers no constraint at all: any twenty-letter sequence is a possible decoding. The more complicated the interpretive scheme, the more text is needed to make its success non-trivial.
- Most structured things say nothing. Molecules, electronic circuits and chess positions are built regularly from standard parts, but very few things can be construed as asserting, asking or commanding.
- Interpret all we can. Interpretation is discovery: finding coherence where it would otherwise go unnoticed. If it is there to be found, there is no further question whether the tokens are "really" interpretable.
- Pluto Potatoes. If potato-like creatures on Pluto have leaf motions that translate, consistently and elegantly, into conversations about us, the weather and astrophysics, we do not ask whether they can "really" talk: discovery of the translation is discovery of the fact.
- Order need not be explained. A consistently workable interpretation establishes that a text is orderly even if nobody can say how or why: discerning the sense is discerning the order.
Reasonable sense
Some ordered texts admit no interpretation: chess games are not random but say nothing. An interpretation works if the text as interpreted makes reasonable sense; rendering the cryptogram as "Zyxw zvut zsu rqppt" (letter salad) or "Vol viny van ruddy" (word salad) is not an interpretation at all. Without the sense requirement any transformation would count.
An enormous stone tablet of hieroglyphs is proposed to be an astronomical calendar. If, so interpreted, it reports several eclipses a night and thirty-seven planets by the North Star, the interpretation is suspect. If it reports eclipses on just the right dates and planets in the right places, it is convincing. So telling the truth is a component of making sense, and random falsehood is a kind of incoherence. But sense exceeds truth: the authors might sensibly omit eclipses visible only elsewhere on the globe, and the tablet could be tax laws, a poetic appeal for healthy children or a list of unanswered scientific questions, none true or false, yet still sense or nonsense depending somehow on the world.
- Truth validates the links. Meanings relate symbols to their objects, and truth discloses and validates those connections.
- Inference from symbols to world. If the calendar is reliably true, you can learn whether the Moon was eclipsed on a date just by reading the symbols.
- Inference from world to symbols. If a piece of the calendar is missing, knowledge of the astronomical facts tells you what it said, again only if the calendar is reliable.
- So: general truth under an interpretation is good reason to assign these meanings rather than others.
Haugeland admits that making sense is philosophically "a difficult and murky notion" that the book never renders transparent; he proceeds on intuition.
Interpreted Formal Systems
Formal tokens can be interpreted: assigned meanings and taken to say things about the outside world. A century ago this was a major innovation with revolutionary consequences for logic and mathematics; if AI is right, the mind is itself a special interpreted formal system.
- Semantics. The general theory of interpretation and symbolic meaning; the meanings assigned, and anything depending on them (reference, truth, falsity, entailment), are semantic properties.
- Syntax. For formal systems "syntactical" just means formal, used when contrasting with semantics (one would not speak of the "syntax" of a chess position, since chess is uninterpreted).
- Two lives. Because formal systems are self-contained, semantic properties cannot be syntactical properties. Interpreted tokens lead two lives: syntactical lives as meaningless markers moved by the rules of a self-contained game, and semantic lives with meanings and relations to the world. The dramatic question is how the two lives get together.
Athol's letter game
Athol finds a game whose tokens are the fifteen letters A to O; a position is a row of letters, a move adds letters on the right, and each starting position has exactly one legal move. He conjectures that the letters are Arabic numerals and arithmetic signs in disguise. There are 1,307,674,368,000 (15!) ways to map fifteen letters onto fifteen digits and signs.
| Game (start, move) | First scheme | Second scheme | Third scheme |
|---|---|---|---|
| OEOA N | =5=1 / | 9/9 = 8 | 1+1 = 2 |
| NIBMA G | /92×1 7 | 83+7 = 1 | 27/3 = 9 |
| HCHCHA KON | 838381 +=/ | 2−2−2 = 598 | 8×8×8 = 512 |
| KEKDOFA F | +5+4=61 6 | 5/5×90 = 0 | 5+5−10 = 0 |
| MMCNA JJ | ××3/1 00 | 77−8 = 44 | 33×2 = 66 |
| OODFA OO | = = 461 = = | 99×0 = 99 | 11−0 = 11 |
| IDLA M | 94−1 × | 3×6 = 2 | 7−4 = 3 |
| NBNA O | /2/1 = | 8+8 = 9 | 2/2 = 1 |
| Verdict | Arithmetic confetti | Look like equations, all wildly false | All true: Athol vindicated |
The first scheme gives gibberish, as would nearly all 1.3 trillion. The second is superficially better (digits and signs in plausible places) but every equation is preposterously false, so it is no real improvement. Only the third yields true equations, and consistently true results make the interpretation persuasive. (Box 5 adds that numerals are not numbers: "five", "cinq", "5", "V" and binary "101" all name one number; binary 101 and Arabic 101 look alike but name different numbers; Latin "sex" means six.)
Axiomatic systems and WFFs
Formal calculation did not revolutionise mathematics; formal axiomatic systems did. Informally, an axiomatic system is a set of axioms (and definitions) plus inference rules for deriving theorems; the axioms are basic and obvious and the rules permit only valid inferences, so derivations are proofs. Euclid is the familiar example, though rarely rigorous; a note records that geometry had no rigorously formal system until David Hilbert produced one in 1899.
- Game one (grammar). Simple tokens combine into allowable complex tokens, the well-formed formulae (WFFs); the legal positions of game one are the tokens of game two.
- Game two (proof). The axiom-WFFs form the starting position; the rules permit adding WFFs depending on what is already there; tokens are never modified or removed, so positions only grow. Each added WFF is a formal theorem and the game leading to it a formal proof.
- Analogy. English grammar decides which word strings are sentences; reasoning then manipulates sentences, as in the syllogism in Barbara: all frogs are mortal; Fido is a frog; therefore Fido is mortal.
- Non-trivial complexity. Which WFF manipulations are legal depends heavily on the internal composition of the WFFs.
For a formal system to be interpretable as an axiomatic system in the intuitive sense, two conditions must hold:
- The axioms should be true (preferably obvious).
- The rules should be truth preserving: a rule is valid if and only if, for any position containing only true WFFs, any new WFF it allows is also true.
Then every WFF in every legal position is guaranteed true; the truth of the axioms is preserved at every step.
The Formalists' Motto
Coherence applies to formal systems with a twist. The "reasonable sense" side works as before (and texts can make sense as questions, stories or poetry, not only as truths, which matters for AI). The "ordered text" side changes because the rules define the entire text (the set of legal games). So the text is not random (the rules are the source of its structure), and since the whole potential text is defined in advance, the interpretation is answerable for all of it, by evidence or proof. Combine these: if syntactic rules specify the texts and the interpretation must make sense of all of them, then simply playing by the rules is a surefire way to make sense. Obey the formal rules of arithmetic and your answers are sure to be true.
FORMALISTS' MOTTO: If you take care of the syntax, the semantics will take care of itself. Taking care of the syntax is just playing by the rules; making sense "takes care of itself". This is how the two lives of an interpreted formal system get together, and the deep reason interpreted formal systems matter at all.
Computers
A computer is an interpreted automatic formal system, that is, a symbol-manipulating machine. Two clauses:
- The referee and at least some players of a formal system have been automated: black boxes that manipulate tokens by the rules.
- The tokens are symbols: interpreted so that legal moves "make reasonable sense" in context.
An automatic formal system literally takes care of the syntax, so by the Motto the semantics takes care of itself automatically. A computer is precisely what Descartes said was impossible: a machine that automatically makes sensible utterances in context.
The pocket calculator is an automated Athol's game with familiar numerals: you enter a starting position, press =, and the machine writes the one legal move, which is arithmetically correct because the interpretation was acceptable only on that condition. Its insides are irrelevant; they could be "mystic glowing jelly" and it would still be a perfectly good calculator.
Two sources of order and the division of semantic labour
Arithmetic is deterministic: the rules fix every move and the player contributes nothing. Sophisticated systems are nondeterministic, so the actual text has two sources of order: the rules' constraints and the players' choices. Interpretation may then find sense at two levels: a basic sense guaranteed by following the rules, and a refined sense in the players' actual choices (interesting theorems, elegant proofs).
In an axiomatised algebra, proving a = a, a + a = a + a, a + a + a = a + a + a, and so on is perfectly valid and gruesomely stupid ("toiling over a dozen or two could win a trip in a padded wagon"). We therefore separate interpreting the system (its theorems are true) from evaluating the player (smart or stupid, sane or crazy). This is the division of semantic labour: rules and axioms account for basic interpretability; players account for refined reasonableness.
- Neither too loose nor too strict. Rules that allow too many moves leave too little order to interpret; rules that allow too few curtail or homogenise the text, as with very short coded messages.
- Formalisation. When rules define a system interpretable as containing only truths, and enough of them, about a mathematical domain, that domain has been formalised. Developing strict formalisation was among the crowning achievements of nineteenth-century mathematics.
- The limit. Outside mathematics and logic, suitable rules are very hard to find: politics, engineering, literature and idle talk resist formalisation or even decent approximation. Fortunately, general formalisation is not required for AI.
Haugeland then takes the division back, or limits it. An ordinary chat with a neighbour coheres in many ways at once:
- Common topics run through groups of utterances.
- Questions and objections are relevant and get answered.
- Surprises get elaborated; banalities are dropped.
- Suggestions are well motivated and apparently feasible.
- Discussion inclines to what is new and interesting to you.
- Evaluations fit the evidence offered.
- Oh yes, and statements tend to be true.
There is no known way to divide this fabric so that one part, enough to support interpretation, is guaranteed by formal rules and the rest credited to the players. For manual systems the division is required, since all legal moves must make sense whoever makes them. Automatic systems are different: tokens, players and referee are built in, so the device's actual behaviour and dispositions form a determinate text that may be interpretable as a whole even if no division exists.
Type A computers. Take an interpreted formal system and automate it, or find formal rules permissive enough to cover every move the machine makes and restrictive enough to support a basic interpretation on their own. The division of semantic labour holds: a base of rule-defined "pure reason" (validity, correctness) with selective wit superimposed, as in the chess machine's lister and chooser. Haugeland suspects science fiction's superrational robots come from seeing only this type.
Type B computers. Systems that do not maintain the division and so are not confined to formalisable domains. A computer conversing in ordinary English would be interpretable only as a whole: the order in the speaker's choices of what to say must be added to the "legal moves" (grammatical sentences) before there is enough structure to interpret. Many if not most contemporary AI systems are generically type B.
LLMs as extreme type-B systems. Large language models fit Haugeland's type B more completely than anything he knew. No set of formal rules about tokens supports a basic interpretation of their output; their text is interpretable only as a whole, by the coherence of what they actually say. Interestingly, the most reliable recent systems re-impose a division of semantic labour where it is available: DeepMind's AlphaGeometry (2024) paired a language model that proposes constructions with a symbolic deduction engine that guarantees validity, solving 25 of 30 olympiad geometry problems in its benchmark, and AlphaProof (2024) generated proofs checked in the formal language Lean, reaching silver-medal standard on that year's International Mathematical Olympiad problems together with AlphaGeometry 2. This neurosymbolic pattern is the Formalists' Motto put to work: let a learned type-B system choose moves, and let formal rules guarantee that every move is sound.
GOFAI
To keep terminology tight, Haugeland capitalises "Artificial Intelligence" and coins a name: Good Old Fashioned Artificial Intelligence, GOFAI for short. Not every intelligent artifact counts. GOFAI rests on Hobbes's idea that ratiocination is computation, and that thesis is empirically substantive: it says something about the world that could be false, and rival accounts are being developed in laboratories. Two consequences follow from the mere possibility of rivals:
- A non-GOFAI artifact. If an alternative theory of intelligence succeeds, one could build an intelligent artifact working as that theory says, and it would not be GOFAI.
- Simulation is not GOFAI. One could also simulate that intelligence on a computer, as scientists simulate hurricanes, protein synthesis, traffic jams and "the black market of Albania". The simulator works by calculating symbolically, often numerically, to describe a system that works by other principles (aerodynamics, not calculation). GOFAI's thesis is not that the processes of intelligence can be described symbolically (hurricanes can) but that they are symbolic. A note credits the distinction to Hubert Dreyfus (1965).
- Our ability to deal with things intelligently is due to our capacity to think about them reasonably (including subconscious thinking).
- Our capacity to think about things reasonably amounts to a faculty for internal "automatic" symbol manipulation.
Two corollaries:
- Corollary 1: worldly interpretation. Insofar as the inner manipulations are intelligent thoughts, they must be interpreted as about the outside world, whatever the system deals with intelligently.
- Corollary 2: at least one level of analysis. An intelligent system must contain computational subsystems ("inner computers") that carry out the reasonable inner manipulations.
What happened to GOFAI. Haugeland's label stuck and became the standard name for symbolic AI. Within a year connectionism (the 1986 Parallel Distributed Processing volumes) offered a prominent non-GOFAI rival, and from 2012 onward deep learning displaced symbolic methods across perception, translation, games and language. His simulation point now applies to most of AI: a neural network running on a digital computer is a formal system at the hardware level, but the computer's symbols are numbers describing weighted activations, not interpreted thoughts about the world. By his own definition, a modern LLM is not GOFAI, even though it runs on a computer and emits symbols.
The homunculus returns, and is tamed
Inner computers sound like homunculi, which threatens the old circle. But the regress argument smuggled in two assumptions: that Zelda has only one homunculus, and that it is as smart as Zelda (or as hard to explain). Neither need be true.
Picture dozens or thousands of homunculi, each relatively narrow-minded, with Zelda's net wit emerging cumulatively from their contributions and organisation, repeated on many levels (a note traces the idea to Jerry Fodor (1968), with related work by Herbert Simon, Daniel Dennett, Haugeland and William Lycan). That is still "more metaphor than science" until we know how there can be any intelligence at all. Enter computers: deterministic systems are easy to automate; the chess machine showed a nondeterministic system automated as a composite of deterministic ones, with a single routine seen as an algorithm or as a fairly clever heuristic; and that structure transfers to a type-A computer, giving a machine with some intelligence. This "nose in the tent" settles the matter on one condition: that homuncular analysis can fill the gap up to full intelligence, including type B. The only standard for "more" and "less" intelligent here is a successful GOFAI analysis itself; until then, hunches. "We won't really know the answer until we really know the answer", but the question is now empirical, and the old metaphysical riddles are cleared away.
- Bob's car. Bob's car dies; he opens the hood, notices a loose wire, reconnects it and drives on. We explain his intelligence by what he saw, knew, inferred and wanted about the car, so many inner symbols must be interpreted as about worldly things.
- Internal semantics. Inner players may also send messages about internal business (who does what when), meaningful but not about the world; in computers, memory addresses and procedure names. Interpreting these is internal semantics.
- Level-relative meaning. A specialist asks the librarian to look up spark plugs; to understand problem solving, the symbol she passes means "spark plug". Analyse the librarian and the same symbol is just a label for the required article. Changing explanatory context reinterprets the symbol, as when one procedure is both heuristic and algorithm.
Is GOFAI too narrow?
Haugeland's definition excludes much research that uses computer models, even some that calls itself AI. He is not legislating for scientists; he is choosing a topic, for four reasons:
- The bulk of work called AI over the previous quarter-century fits the definition (though some makes no psychological claims).
- GOFAI is widely regarded as the most worked-out, theoretically powerful and practically successful approach psychology has ever had.
- There are intuitive, long-standing positive reasons to find it attractive.
- The more focused the topic, the easier it is to say true and interesting things.
None of this means GOFAI is right, only that it is worth discussing.
Paradox and Mystery Dispelled
A GOFAI system has an inner playing field where inner players, under an inner referee, manipulate inner tokens; these manipulations are interpreted as the thought processes by which the whole system makes sense and acts intelligently. Overall intelligence is explained by analysis into less intelligent components whose external symbolic moves are the larger system's internal reasonable cognitions. That is the paradigm of cognitive science.
Dissolving the paradox of mechanical reason
Dimming homunculi defuse the infinite regress, but not the paradox itself (attend to meanings and you are not mechanical; ignore them and you are not reasoning). Intelligence comes in degrees, but meaningfulness does not: tokens cannot become "less and less symbolic"; either they are interpreted or they are not. A different strategy is needed: redescription, seeing the same thing from radically different points of view, with a basis in the phenomena for each. Interpretation is redescription grounded in the coherence of a text, and for computers its foundation is the Formalists' Motto.
The paradox is resolved by assigning its two horns to two modes of description of the very same process. Described one way, the inner players are automatic formal systems manipulating tokens by rules, all quite mechanically; described another way, the same players manipulate the same tokens, now interpreted as symbols, reasonably in accord with what they mean. "Computational homunculi can have their meanings and eat them too."
This works as stated only for type-A systems, because the Motto presupposes the division of semantic labour. So Haugeland goes one more round with analysis. Imagine Pierre, analysed without remainder into inner players, referee and tokens, each known to be fully mechanical from the right point of view; then Pierre is fully mechanical too. Yet the same inner players, seen differently, may manipulate his inner symbols intelligently, contributing more meaningful order than any formal rules impose, order that may be needed for his inner text to be interpretable at all. Then Pierre is a type-B computer, still redescribable as fully mechanical via the analysis. Whether all type-B systems (native English speakers, say) are so analysable is "bread and butter, workaday AI", an empirical question: "mechanical reason is paradoxical no more."
The mystery of original meaning
Hobbes's problem of telling minds from books is older and deeper. Meaning in this chapter rests on interpretation and coherence, which works the same for inscriptions, codes, foreign languages, formal systems and (presumably) brains, and is neutral between original and derivative meaning. Interpreting brain symbols with original meaning would be discovery (cracking someone's code) rather than stipulation (giving a new notation its meaning), but the canon of success is the same. So the questions remain: which symbolic systems mean originally, and why?
- Activity alone is not enough. Books just sit there while thoughts interact, but print words on thousands of floating magnets or the backs of busy ants and the symbols interact vigorously while their meanings stay just as derivative.
- Semantically appropriate activity. Write "All frogs eat insects" and "Fido is a frog" on paper and nothing happens; put them on magnets or ants and the activity has nothing to do with frogs. Put them into someone's head as beliefs, and very likely a new symbol appears: "Fido eats insects", a valid conclusion. That is why we think people understand their thoughts and paper does not.
Haugeland's distinction: thought symbols are semantically active, because their changes and interactions are semantically appropriate; printed or magnetic symbols are semantically inert, mattering as meaningful things only when users take them so, which is why their meaning is derivative. Semantically active symbols are prima facie candidates for original meaning, since they interact appropriately independently of any outside user.
"Active" is a literary conceit: such symbols are interpreted states of an integrated system in which they are manipulated automatically, with the brain processes hidden backstage. But that describes a computer: interpreted tokens interacting automatically and appropriately with no outside user. Is the mystery solved? Haugeland compares two extremes.
The pocket calculator. It manipulates numerals appropriately (always the right answer), yet it is hard to say it refers to numbers by itself. It "has no idea" what numbers are: it can't count, can't explain, "can't tell a proper fraction from a buffalo chip", and doesn't care; it cranks four mindless algorithms. Automatic bank tellers, word processors and many computer games belong with it.
The robot civilisation. GOFAI systems "of fondest fiction" in mobile, versatile bodies, capable of human-range communication, problem solving, artistry and heroism, populate an Earth where humans have died out; they build cities, do science, fight lawsuits, write volumes, and a few in ivory towers wonder how their minds differ from books. Insisting their treatises and songs mean exactly nothing "just seems perverse": if GOFAI can get this far, such systems can have original meaning.
- Continuous spectrum. Calculators have minimal semantic activity and minimal original meaning; super robots have a full complement of each.
- Spectrum with a threshold. Original meaning appears only above a minimum level of semantic activity. (Both 1 and 2 make originality a matter of degree and slide down a dubious slippery slope.)
- Special kinds. Only semantic activity of certain sophisticated kinds or with distinctive characteristics yields original meaning, raising the questions what kinds, why, and can GOFAI build them (Chapters 5 and 6 take up the first two).
- Flat rejection. Perhaps the robot scenario is impossible, and something quite different from semantic activity is needed for original meaning.
The mystery is "battered but not down." Semantic activity and automatic computation make it far less mysterious, but what exactly makes the difference, and why, remains unsettled without further conceptual and factual tools, which the next chapters on computer architecture and actual AI research supply.
Original meaning and grounding in the LLM era. The debate Haugeland framed is now central. Stevan Harnad's "symbol grounding problem" (1990) sharpened Hobbes's worry: symbols defined only by other symbols are like a dictionary in a language you cannot read. Bender and Koller (2020) argued that models trained on form alone cannot acquire meaning; defenders point to multimodal training, interaction with tools and environments, and use-based theories of meaning. Interpretability research gives Haugeland's "semantically active" criterion some empirical bite: a transformer trained only on Othello move sequences was found to build an internal representation of the board (Li and colleagues, 2023), and dictionary-learning studies of production models (Anthropic's "Scaling Monosemanticity", 2024) recovered millions of interpretable internal features, such as one for the Golden Gate Bridge, that causally steer outputs when amplified. Whether such internal states have original meaning or only very sophisticated derivative meaning inherited from human text remains unresolved, which is roughly where Haugeland left it.
Fido in the language model. Give a modern chatbot "All frogs eat insects. Fido is a frog." with no question attached. It will very likely volunteer "So Fido eats insects." By Haugeland's test, its internal tokens change in a semantically appropriate way without any user manipulating them, so they look semantically active, unlike ink or ants. Now ask: is it the calculator (an elaborate engine that "has no idea" what frogs are) or the robot philosopher? The experiment shows that semantic activity is cheap to observe and hard to evaluate: Haugeland's criterion picks out candidates for original meaning but cannot, by itself, settle which candidates succeed, which is exactly why he leaves options 3 and 4 open.
Is coherence enough? Haugeland's account makes meaning a matter of interpretability by an outside interpreter, which some critics argue simply relocates derivative meaning (the interpreter's sense of "reasonable sense" does the work). His own later writing moved toward the view that genuine understanding requires commitment and responsibility to how things are, something a merely coherent symbol system may lack. The chapter's "paradox dispelled" is also narrower than its title: it shows mechanism and reason are compatible under redescription, not that any actual mechanism reasons.
For enterprise AI, Haugeland's division of semantic labour is a design principle. Wherever a domain can be formalised (calculations, eligibility rules, regulatory thresholds, reconciliations), keep a rule-based layer whose outputs are guaranteed by construction, and let a learned model contribute only the choices (which case to escalate, which explanation to draft). Where no formalisation exists (relationship-manager conversations, narrative risk summaries), the system is type B and can be validated only as a whole, by the coherence and truth of its actual outputs: hence sampling, human review and outcome monitoring, not proofs.
Computer Architecture
People assume all computers are essentially alike, and so imagine only one design for AI or the mind. Haugeland tours five radically different architectures (Babbage's Analytical Engine, the Turing machine, the von Neumann machine, McCarthy's LISP and Newell's production systems), each universal, each with its own memory organisation and control structure. The payoff is a single claim: the mind could have a computational architecture all its own, so mental architecture becomes an empirical variable for cognitive science to discover.
Babbage and His Analytical Engine
The point of the tour is not to nominate a psychological design but to show how much machines can vary and what the theory looks like. Haugeland starts with the prehistory of the calculator.
- c. 1623 Wilhelm Schickard (1592-1635), an obscure German astronomer, invents the automatic digital calculator; nobody knew of it until recently, and "who knows what else may still be lost".
- c. 1640s Blaise Pascal (1623-1662) builds the first publicised calculator; it causes a stir though it can only add and subtract, and clumsily.
- 1670s Gottfried Leibniz (1646-1716) adds multiplication and division; then nothing exciting happens for two centuries.
- c. 1823-1833 Charles Babbage designs the Difference Engine: prototypes and parts built, never completed.
- c. 1833 onward Babbage turns to the Analytical Engine, his life's crowning achievement, also never finished.
- 1842 L. F. Menabrea, an Italian engineer, publishes (in French, in a Swiss journal) a summary of Babbage's private lectures in Naples.
- 1843 Ada Augusta (later Countess of Lovelace) translates it into English, adding notes three times longer and more sophisticated than the original.
Charles Babbage (1792-1871 in Haugeland's dates) was history's first computer scientist and a classic eccentric genius, notorious for an obsessive hatred of organ grinders. Trained as a mathematician, he held Newton's Lucasian chair at Cambridge without ever teaching a course or moving from London, and made original contributions to precision metalworking (his shop was among Europe's finest) and to what is now operations research.
- The Difference Engine failed for three reasons. Critical tolerances made construction very expensive; Babbage kept improving the design, forcing constant rebuilding; and around 1833 a grander scheme seized his imagination.
- The Analytical Engine's legacy. It was never finished either, but Babbage left nearly a thousand technical drawings, timing diagrams and flow charts and six or seven thousand pages of notes; a substantial model was under construction when he died at seventy-nine.
- Lovelace's standing. Babbage probably supplied some of her material, but Lady Lovelace understood his work better than anyone else in his lifetime; the programming language Ada is named for her.
The Analytical Engine incorporates two profound and unprecedented ideas, which together are the foundation of all computer science: (1) its operations are fully programmable, and (2) its programs can contain conditional branches.
Mill, store and referee
The Engine has three components: the mill (an arithmetic unit), the store (data memory), and an unnamed sequencer/controller. Its tokens are signed numerals and its playing field is essentially the store. In the vocabulary of earlier chapters, the mill is a team of four narrow inner players, one per arithmetic operation, and the controller is the inner referee: it never touches tokens itself, but calls on players one at a time, saying which numerals to use and where to put the answer. Every move is fully determined.
What game does it play? Whatever game you tell it to, within limits: hand the referee different rule "definitions" and the Engine becomes any automatic formal system you like. Making a specific automatic system simply by describing it appropriately to a general-purpose system is programming, "one of the most powerful ideas in the history of technology", and Babbage invented it all by himself.
- Against the calculator. A pocket calculator does the same four operations but you enter each command by hand: to evaluate 3(x + y) − 5 you add, multiply by three and subtract five, and repeat the whole sequence for every new x and y. It is really four independent special-purpose devices.
- The better plan. A machine for which you can prespecify any sequence of basic operations on specified variables can evaluate any formula, and given the specification it is the special-purpose machine for that formula until the specification changes.
- The conditional branch. A directive to the referee to move to another part of the program depending on the result of a test. Babbage's programs are branched schedules, like the key/lock algorithms of Chapter 2, and conditional control matters equally in the symbol manipulators of AI.
The user programming facility is the least well-defined part of Babbage's design, so Haugeland gives a compromise notation. Store locations are named V1, V2, V3...; "V1 + V2 → V3" means add the contents of V1 and V2 and put the sum in V3. Conditionals test whether a number is greater than zero and, if so, move the referee a relative number of lines (not to a named line).
The program computes a·r^m + r^n. Load V1 = a, V2 = r, V3 = m, V4 = n, and V5, V6, V7 = 1.
- Loop one. V5 × V2 → V5; V3 − V7 → V3; IF V3 > 0, GO BACK 2 LINES. After m passes, V5 holds r^m.
- Coefficient. V1 × V5 → V5 gives a·r^m.
- Loop two. V6 × V2 → V6; V4 − V7 → V4; IF V4 > 0, GO BACK 2 LINES, giving r^n.
- Finish. V5 + V6 → V8; PRINT V8; STOP.
Brass and cards
As an automatic formal system the Engine is strictly medium independent, but 150 years ago physical realisation was the obstacle, and the design is astonishing.
- Forty decimal digits plus a sign per numeral (not binary); several ideas for fractions, none settled.
- One brass gear per digit, roughly the size of a doughnut, forty stacked a few inches apart on a brass rod about ten feet high.
- More than a hundred columns in the mill, arranged in a circle four or five feet across.
- About a hundred more columns in the store, in a double row to one side.
- Size of a small railway locomotive, though far more complicated and expensive; recent analyses suggest Babbage's parts were accurate and strong enough for a working Engine.
Programs were not kept in the store but encoded on punched cards strung together with ribbons, which let the machine back up and repeat a group of instructions, essential for loops. The arrangement came from the cards controlling the brocaded patterns of Jacquard looms, which inspired Lovelace's famous simile: the Engine "weaves algebraic patterns just as the Jacquard-loom weaves flowers and leaves." And just so, if AI is right, does a mind weave general symbolic patterns.
Small slips in the potted biography. Babbage is now usually dated 1791-1871 (born 26 December 1791), and his Lucasian professorship is usually given as 1828 to 1839, nearer eleven years than nine. Neither affects the argument. Lovelace's notes are also now famous for Note G, a step-by-step plan for computing Bernoulli numbers on the Engine, often called the first published computer program; historians still argue over how much of it was hers and how much Babbage's, which is roughly Haugeland's own hedge.
Babbage vindicated in brass. London's Science Museum completed the calculating section of Babbage's Difference Engine No. 2 in 1991 (the printer followed in 2002), built to his drawings with tolerances achievable in his day, and it works, confirming Haugeland's report that the parts were good enough. The Analytical Engine itself has still never been built; a volunteer effort, Plan 28, launched in 2010 by programmer John Graham-Cumming, has been digitising and studying Babbage's archive with the aim of eventually doing so.
Turing Machines and Universality
Alan Turing (1912-1954) was prominent on three counts: the 1950 Turing test, major contributions to pioneering British electronic computers during and after the war, and, most important, the first mathematically sophisticated theory of computation, produced before the war, including "astonishing and profound discoveries" about machine capabilities. Central to it was a new basic architecture: the Turing machine. There are infinitely many possible specific machines, none very significant alone; what matters are the category and the general theorems about it.
Turing never built one either, for opposite reasons. The Analytical Engine was near the limits of what could be understood, let alone built; Turing machines are amazingly simple. Babbage wanted practical use; Turing wanted abstract theory, so his designs are elegant to describe, convenient to prove things about, and utterly impractical to use: building one would be easy and pointless.
- The tape. A passive storage medium divided into squares, each holding one token from a finite alphabet; unlimited in length, but only a finite segment is occupied, the rest blank. It also serves for input and output: write tokens before starting, read what remains after halting.
- The head. The active part, moving one square at a time, scanning a single square it can read or write, and always in one internal state from a finite repertoire; a special halt state stops the machine.
- Two factors decide each step: (1) the token in the scanned square and (2) the current internal state.
- They decide three consequences: (1) what token to write on the square, (2) which square to scan next (same, left or right), and (3) which state to enter (or halt).
So a Turing machine's entire functioning fits in one two-dimensional chart: a row per token, a column per state, each entry giving the three actions. As an automatic game, the tape is the playing field, the internal states are very narrow inner players (each reads the token, replaces it, tells the referee which way to move and which player to call next), and the referee is "even dimmer": he starts with player 1 and calls whoever is requested until someone says HALT.
Chart entries are three characters: token to write, L or R, next state (or HALT).
- ARTHUR (alphabet A, B, C plus blank) starts in state 1 inside a string. In state 1 he moves left, changing nothing; at the first blank he switches to state 2 and moves right, swapping every A for B and every B for A while leaving C alone; at the right end he halts.
- BERTHA (A's and asterisks, nine states) runs to the right end of the string counting A's. One or no A's: halt. An even number: go back and turn every second A into an asterisk, then start over. An odd number: add three more A's at the right end, then start over.
- The punchline. If the number of A's is not a multiple of 3, BERTHA methodically eliminates all but one and halts gracefully; if it is a multiple of 3, she lands in an infinite loop and "never rests".
Will BERTHA stop? Halving keeps a multiple of 3 a multiple of 3, and adding 3 does too, so BERTHA can never reach one A from a multiple of 3: a few minutes of arithmetic predicts her fate. Now change her rule from "add three" to "triple and add one". The machine is just as small, but whether it halts for every starting string is the Collatz conjecture, unsolved since 1937. The exercise shows what Turing proved in general in 1936: there is no algorithm that decides, for every machine and input, whether it halts. Tiny charts can hide questions no mathematician can answer, which is one reason "it only does what it's programmed to do" explains so little.
Turing's thesis
Recall finite playability: a formal system's rules must be "followable by players that are indisputably finite", a vague condition. In the mid-1930s several mathematicians proposed rigorous, intuitively plausible criteria, and all were soon proved equivalent, so the plausibility of each supports all the others; every plausible criterion proposed since has also proved equivalent. Essentially all mathematicians now accept that these criteria capture finite playability. One formulation:
TURING'S THESIS: For any deterministic automatic formal system whatever, there exists a formally equivalent Turing machine. No automatic formal system can do anything (nonrandom) that Turing machines cannot; it is a thesis, not a theorem, because it is a proposal about what "finite playability" really means and cannot be proved.
Programming and the universal machine
Some Turing machines can mimic others. Encode a description of the machine to be imitated as part of the imitator's input; the imitator reads it, computes a lot, and from time to time makes a move in a reserved part of its tape exactly equivalent (up to transliteration) to the imitated machine's next move. The encoded description functions as a program.
- MOM and BABY. MOM's tape has squares tinted alternately pink and light blue (for our convenience; MOM is colour-blind). BABY's chart, stretched into one line with punctuation, is written on the pink squares; the blue squares stand in for BABY's tape. MOM tracks BABY's position and state on spare pink squares, reads the relevant blue square, looks up BABY's response and does it.
- SUPER-MOMs. Turing proved more: TURING'S PROOF: there exist universal Turing machines that can mimic, move for move, any other Turing machine whatever. With one universal machine "and divine patience" you never need another: code up the chart, transliterate, and watch only the blue squares.
Thesis and proof complement each other splendidly: by the thesis, Turing machines are the only automatic formal systems we need; by the proof, we need only one, a universal one. One SUPER-MOM can do the work of all possible deterministic automatic formal systems. Universal machines need not be complicated, only slow: the program is just primitive rules in a regimented form, so MOM needs a few basic abilities to decode and follow rules, and lots of time.
- Minsky's universal machine. The widely accepted measure of a Turing machine's simplicity is states × symbols, since tricks can trade one for the other.
- 7 states × 4 symbols = 28: Marvin Minsky's universal machine, the simplest known in 1985; its proof of universality is "fairly difficult and lengthy" (Minsky 1967, section 14.8), and Haugeland prints the chart "purely as a work of art".
Smaller universal machines. The record has fallen since. Yurii Rogozhin and later Turlough Neary and Damien Woods found standard universal machines with products of about 24, and in 2007 Alex Smith won a prize offered by Stephen Wolfram by proving that a 2-state, 3-symbol machine (product 6) is universal, under a looser definition that allows an infinite, non-blank but simply structured starting tape. The disputes over that definition illustrate Haugeland's caveat that "simplest" is hard to define. Turing's general lesson is untouched: universality is cheap, speed and convenience are what distinguish real architectures.
Since Turing's proof, many other universal architectures have been found, including Babbage's. The remainder of the chapter examines three more, each different from the others and from Turing machines, and each widely used in practice.
The Ubiquitous von Neumann Machine
John von Neumann (1903-1957), born and educated in Hungary, moved to the United States as a young man, and his move symbolises three transitions in the story: from European to American characters and US domination of computing after the Second World War; to the earliest practical general-purpose computers; and to computer science as a mushrooming collective enterprise, where crediting ideas to individuals becomes tenuous. Babbage was a lonesome hero; Turing machines are one man's invention; "the von Neumann architecture" evolved over years with inseparable contributions. Von Neumann remained primarily a mathematician, while doing original work in quantum mechanics and economics (the theory of games). He came to computers almost accidentally as a consultant on ENIAC, then the largest computer in the world; reflection on ENIAC's weaknesses led to the IAS (Institute for Advanced Study) computer, smaller but more powerful and much easier to program, the basis of many later designs.
Two ways to reach memory
The core is a large memory accessible in two ways:
- Relative access. Specify the next location by its relation to the current one, like directions to a stranger: "down two blocks, make a left, second door on your right". It needs a regular organisation (like the linear Turing tape) and a current location. A Turing tape is accessible only relatively.
- Absolute (random) access. Specify a location by a unique name or number, like a postal address (1234 Main Street, or the Cathedral of Learning, Pittsburgh) or a phone number. It needs no structure and no current location. Babbage's store is accessible only absolutely: its columns have names (numerals) with no usable relations.
Two jobs for one memory
The von Neumann main memory is accessible both ways and also does two jobs, which turns out to matter a great deal:
- It holds the data: initial values, intermediate results and final output (for a sorting program: the original list, partly sorted lists, the final list). It is like Babbage's store but allows relative access, useful for structured data: go absolutely to the start of a list, then move relatively item by item.
- It holds the program. Von Neumann programs are branched schedules like Babbage's, but stored in the same memory as the variables, not on cards. The program is usually accessed relatively (step by step, or back and forth in loops).
Subroutines: the real advance
The architecture "really shines" in using absolute access to implement subroutines: mini-programs a larger program can invoke at any point as if they were a single step. A record-keeping system that needs to alphabetise lists at various times should not carry a copy of the alphabetiser at every point of use; it keeps one copy at an agreed location and branches to it, and the subroutine branches back when done.
The trick lies in the return branch: the subroutine may be called from anywhere, so how does it know where to go back? Put the caller's current address in a prearranged location before branching; when finished, the subroutine fetches it and branches back absolutely.
Why Babbage cannot do it. His program cards could be accessed only relatively (advanced or backed up so many cards), but subroutine calls need absolute access. And return addresses differ from call to call, so they must be rewritable by the machine during the run; Babbage's cards were permanently punched, changeable only by punching and stringing new ones by hand. ENIAC was like Babbage in this respect.
Why von Neumann can. The machine can read, write and use addresses stored in main memory, which it can alter while the program runs. So absolute subroutine calls and returns are feasible and convenient: for Haugeland, the primary architectural advance of von Neumann's design.
- One shared slot. If there is only one prearranged return location, only one subroutine can be active, and no subroutine can call a sub-subroutine.
- One slot per subroutine. The standard approach in von Neumann's day lets each find its way back independently, but a subroutine still cannot call itself (recursion), which sounds bizarre but is feasible and useful.
- The stack. The general solution, invented in the mid-fifties by Allen Newell and Cliff Shaw, is a last-in-first-out memory, like a pile of letters answered from the top (bad for correspondents: "the poor guy who wrote you first gets answered last"). A call pushes its return address; a finishing subroutine pops the top address and branches there. Every subroutine returns to the most recent unanswered call, so calls can nest arbitrarily. The stack itself uses a particular sort of relative access.
In the game picture, the von Neumann referee has real work: he tells players whose turn it is and which tokens to use. Branches, including calls and returns, are not token manipulations by players but shifts by the referee in deciding whom to call next, so the stack is the referee's private memory.
Modularity, libraries and virtual machines
Subroutines are efficient and also modular: once written and debugged, a subroutine can be used anywhere without further ado, so a large program breaks into manageable subproblems, and debugged subroutines can be borrowed wholesale by other programs.
Higher-level machines ("programming languages") are libraries of subroutines plus an automatic librarian. Collect widely useful subroutines, add a referee to handle calls and bookkeeping, and the subroutines become new primitive operations, new players in a fancier game: "in effect, you have a new and more powerful computer." This is the essence of BASIC, Pascal, FORTRAN and COBOL.
- Coincidental architecture. These virtual machines happen to be roughly von Neumann (branched schedules, loops, named subroutines), but that is a coincidence: one could as easily write a von Neumann program that exactly simulates a Babbage or Turing machine, and that would then be the machine you work with.
- Universality that matters. Von Neumann machines (with unlimited memory) are universal in exactly Turing's sense, but their universality is far more important in the real world: they are much easier to program, much faster (no shuttling along a tape one square at a time), and easy to build from basic electronic components, unlike some other universal architectures that are easy to program and fast enough but hard to build.
- The upshot. Virtually all programmable computers in production have a basically von Neumann architecture at the hardware level, and the AI architectures below are, "at present anyway", simulated on von Neumann SUPER-MOMs.
Credit and emphasis. Most historians identify the defining von Neumann idea as the stored program (instructions held in the same rewritable memory as data), described in his 1945 "First Draft of a Report on the EDVAC", whose sole authorship angered J. Presper Eckert and John Mauchly, ENIAC's designers, who had discussed stored programs earlier. Haugeland's emphasis on machine-alterable addresses for subroutine calls is a consequence of that idea rather than a rival to it. On the stack, Newell and Shaw's IPL list-processing languages did pioneer push-down lists in the mid-1950s, but Turing's 1946 ACE design already used a last-in-first-out store for subroutine return addresses (his "BURY" and "UNBURY" operations), and Friedrich Bauer and Klaus Samelson patented a stack principle in the late 1950s, so the invention has several parents. Two smaller points: McCarthy proposed time-sharing in a 1959 memo, but the first working systems were built by others, notably Fernando Corbató's CTSS at MIT (1961); and Turing wrote his 1936 paper as a young fellow of King's College, Cambridge, just before leaving for doctoral study at Princeton. The pioneers Haugeland lists as living have since died: Newell in 1992, McCarthy in 2011, Minsky in 2016.
McCarthy's LISP
John McCarthy (born 1927) is the first native American in the survey and the last trained mathematician. By 1956 he had coined the term "Artificial Intelligence" and organised the field's first scientific conference; in 1958 he and Marvin Minsky founded the MIT AI Lab, and in 1963 he set up another at Stanford, both world centres. He also developed the first time-sharing systems, letting one central computer serve many users at once. The topic here is his 1959 invention of a new kind of computer, LISP: "McCarthy machines", simple and elegant, and distinctive in two respects, memory organisation and control structure.
Memory: from tin cups to Y-connectors
Picture tin cups on a chain, one atomic token per cup. If cups are reachable only by relative position, one step at a time, it is a Turing tape; if each also has a unique label and the processor can jump anywhere, it is a von Neumann memory. Both share a structure that is:
- LINEAR: the chain has no forks or junctions;
- UNITARY: there is only one chain;
- FIXED in advance: the chain is always the same.
None of these is essential to computer memory. Replace the cups with a heap of Y-connectors (a base plug and two angled sockets, like the gadget for two light bulbs in one fixture). Plug bases into sockets to build Y-connector trees, each with one exposed base, its root, label the roots, plug atomic tokens into all the empty sockets as leaves, and you have LISP memory.
- Nodes. The joints are nodes; the atoms in terminal sockets are terminal nodes; a single atom is itself a one-node "seedling" tree.
- No empty sockets. The special atom NIL serves as "blank" or "nothing".
- Asymmetry. Unlike real Y-connectors, LISP nodes have explicitly left and right branches, so any node can be reached by absolute plus relative access: name the tree, then give a path of left and right turns from the root.
- Lists. Line up one Y-connector per item, plug each into the right socket of its predecessor, put the items in the left sockets in order, and NIL in the last right socket. A list of lists is just as easy.
- Not linear. Trees branch, though lists can stand in for linear structures when needed.
- Not unitary. Many trees are alive at once.
- Not fixed. Trees are built to order as needed and thrown back on the heap afterwards, with two consequences.
- Shape is remembered. Since trees differ in shape, the connections themselves are stored, not just the contents, so trees are especially convenient for complex structural relationships.
- No advance planning. Nobody needs to know beforehand how big or what shape the structures will be; details are settled on the spot, so LISP programs can be extraordinarily flexible.
Control: two kitchens
Control is determining who does what when. Order breakfast: chilled orange juice and two fried eggs. The primitive chefs are not the issue; the executive chef, the kitchen referee, is.
- Von Neumann's kitchen. The referee follows a schedule: ORANGE; JUICE that orange; CHILL that juiced stuff; EGG; EGG; CRACK those eggs; SKILLET; FRY those cracked things in that skillet; SERVE. The routine is linear and ordered by priority in time.
- McCarthy's kitchen. The same actions in the same order, but arranged from the opposite end. The referee starts from the required result and works back to prerequisites: she tells SERVE to serve, SERVE needs juice from CHILL and eggs from FRY, CHILL needs JUICE, JUICE needs ORANGE, and so on. Everyone passes the buck until ORANGE, "a simple stockboy who requires no preparations", hands over an orange. Then the referee, remembering who wanted what, passes results back up until SERVE brings breakfast.
The structure is hierarchical, not linear, ordered by results and prerequisites, not by time: engaging runs from the goal down, delivery runs back up.
- Functions are specialists of this kind; their prerequisites are arguments and their results values; functions with no arguments are constants, always returning the same value.
- FUNCTIONS: for each allowable argument (or combination of arguments), there is one and only one (permanently assigned) value. MOTHER[x] qualifies (everyone has exactly one mother); UNCLE[x] does not (some people have none, others many).
- LISP's two differences. LISP functions are active players that evaluate themselves when called, and their uninterpreted arguments and values must always be LISP trees.
- Arguments are evaluated first. In THIRD LETTER OF [SECOND WORD OF [THIS BOOK'S TITLE]], the argument is not the expression "SECOND WORD OF..." (third letter C) nor the word "BOOK'S" (third letter O): it is the value of the inner function, whose own argument is the constant THIS BOOK'S TITLE, evaluated straightaway. The answer is T (from "Intelligence").
Programming by definition
You program a LISP machine not with a schedule but by defining new functions in terms of existing ones, as in algebra: SQUARE[x] =def TIMES[x, x], or a BREAKFAST function defined from SERVE, FRY, SKILLET and the rest. Defined functions are "management" specialists: they return values entirely by telling the referee whom else to call. Real LISPs offer many convenient primitives, but quintessential LISP needs only six, all taking and returning trees:
| Primitive | What it returns | Role |
|---|---|---|
| LEFT[x] | Whatever is in the left socket of the base node of x; error if x is an atom | Disassemble trees |
| RIGHT[x] | The right half of x, likewise | Disassemble trees |
| JOIN[x,y] | A new Y-connector with x in the left socket and y in the right | Assemble trees |
| EQUAL[x,y] | TRUE if x and y are the same, otherwise FALSE | Test |
| ATOM[x] | TRUE if x is a single atomic token, otherwise FALSE | Test |
| IF[x,y,z] | The value of y if x is TRUE, of z if FALSE (evaluating only one) | Choose what gets evaluated |
With these six one can build, modify and destroy trees of arbitrary complexity in any algorithmically specifiable way, so LISP is universal. IF does not "branch" around a schedule (there is none); it decides which functions actually get evaluated. (In McCarthy's own notation these are CAR, CDR, CONS, EQ, ATOM and the conditional COND; Haugeland renames them for readability.)
LISP's memory and control share "a beautiful inner unity": functions applied to arguments that are themselves functions form a tree. LISP writes a function call as a list (function name first, arguments after), arguments are themselves calls, and so the whole is a list of lists of lists, which is to say a tree. LISP programs and LISP data structures have exactly the same underlying form.
To be universal, LISP must repeat a process until a condition is met. Von Neumann machines use loops and conditional branches; McCarthy machines use recursive definitions, in which a function is used in its own definition provided there is an escape clause: a definition may be "circular" "as long as it's not circular forever."
COUNTATOMS[x] =def IF[ATOM[x], (THEN) 1, (ELSE) COUNTATOMS[LEFT[x]] + COUNTATOMS[RIGHT[x]]].
If the tree is an atom, return 1; otherwise call yourself on each half and add. Two things keep the circle from being vicious: each self-call has a simpler argument, and arguments eventually become so simple (atoms) that the function answers directly. Then sums flow back up until the original call gets its value.
LISP after 1985. Dedicated LISP machines (from Symbolics, LISP Machines Inc., Texas Instruments and Xerox) were built in the early 1980s, but the market collapsed in the late 1980s as general-purpose workstations caught up and the expert-systems boom faded, part of the second "AI winter". LISP itself survives in Common Lisp, Scheme, Racket and Clojure, and its ideas went mainstream: garbage collection, conditional expressions, recursion, functions as values and the read-eval-print loop are standard in Python, JavaScript and most modern languages. The "code is data" unity Haugeland admires lives on in macros and in tree-structured formats such as JSON and abstract syntax trees. AI itself moved to Python as glue around numerical libraries, because the heavy computation in deep learning is matrix arithmetic, not tree manipulation.
Newell's Production Systems
Allen Newell (born 1927) considered a mathematics career, then chose the Rand Corporation, a California think tank, over graduate school in New Jersey. There he met Cliff Shaw, a farsighted computer pioneer who joined Rand the same year (1950), and Herbert Simon, a rising star in economics and management science visiting from Carnegie Tech in summer 1952. By 1955 the three had more or less invented general symbol manipulators, heuristic search and Artificial Intelligence. In the 1960s, after Simon lured him back to academia, Newell turned to production systems, "Newell machines".
Chefs who watch the counter
The players are productions. Like Turing and von Neumann players (unlike LISP functions) they act on a common linear memory, the workspace, but they decide who works where when differently: each production constantly surveys the whole workspace for its particular pattern and, when it finds it, goes there and performs its specialty. One modest expert waits for a pint of whipping cream flanked by a whisk and a cold copper bowl; when that appears he whips the cream and discards the whisk, which may satisfy the cake decorator waiting for a bowl of whipped cream.
- Content-based access. Locations are found neither absolutely (by name) nor relatively (from here) but by their contents: not "1234 Main Street" or "six blocks north and two east" but "a yellow brick split-level with pink curtains, wherever it may be." The described contents are just the patterns the productions seek.
- Self-activation. Nobody tells productions when to act; they wait until conditions are ripe. Turing units are nominated by predecessors, von Neumann operations are prescheduled, LISP functions are invoked by other functions; productions act on their own. There is no central control, productions never interact directly, and all communication is via "to whom it may concern" notices on a public bulletin board. Direction emerges as the net effect of independent decisions, like Adam Smith's "invisible hand" in an open market.
Two loose ends need a referee. Conflict resolution: if two productions are satisfied at once, the simplest rule is to rank-order them and pick the highest-priority petitioner, as though they stood in line, the first checking for his pattern, then the second, and starting over at the head of the line after any production acts. Multiple matches: if a pattern occurs twice, use a prior ordering, such as scanning left to right and taking the first.
Programming by productions
An unprogrammed production system is an empty workspace, a referee and a kit of primitives; the productions themselves are the program. Each production has a condition (a pattern that might appear in the workspace) and an action (usually modifying that pattern, sometimes external output):
WHENEVER (condition) IS SATISFIED, PERFORM (action), schematically condition → action. The primitive abilities needed are pattern recognition and modification of the patterns recognised.
An algebra simplifier might contain "A + B = C + B" → replace with "A = C", where A, B and C stand for arbitrary strings of tokens. Then 2x + 4y(z − 14) = w − 3 + 4y(z − 14) reduces to 2x = w − 3 in one step. That presupposes "fairly classy" matching (recognising that the complex 4y(z − 14) fits B on both sides), but such facilities are common and make productions easy to design and understand.
- A throwback to Turing. Like Turing machines, and unlike von Neumann and LISP machines, Newell machines have no explicit IF: every step is conditional. Von Neumann and McCarthy machines have a natural order of execution built in (follow the schedule; evaluate arguments before functions), so conditionals are needed only occasionally; Turing and Newell machines have no predisposed flow, so a decision is made at each step.
- But a different spirit. Turing made the machine as simple as possible and let programs be "numbingly large and tedious"; production systems have powerful primitives (automatic finding and matching of complex patterns with variables), so programs are brief and elegant: each line does a lot.
Production systems are universal, so they must repeat; in fact repetition comes naturally and the hard part is stopping it.
- One production: any integer N → increment N. Given a number, it increments "till the cows come home", with no loop, branch or recursion telling it to.
- Stopping. Add a higher-priority production, any integer greater than 99 → stop (or build the test into the original condition).
- COUNTATOMS again. With a tree-holding workspace and a counter starting at 0: (1) any atomic tree A and integer N → delete A and increment N; (2) any tree T → replace with LEFT[T] & RIGHT[T]; (3) anything at all → stop. Production 2 chops trees in half, production 1 sweeps up atoms and counts them, and production 3 gets its turn only when nothing else can act. The looping "just takes care of itself", the invisible hand again.
Modularity and its limits
A module is an independent subsystem with a well-defined task and narrowly defined interactions, like the tape deck and preamp of a component stereo. LISP's user-defined functions are modules, but less independent than productions: a function must be called explicitly, and the caller (really the programmer) must know when to call it, its name, its arguments and what it returns, for every situation where it might be needed.
A robot taxi driver is complete except that its designers forgot about sirens: whenever it hears one, it should pull over until the danger passes. The new unit checks memory for a SIREN report from the "ears" and outputs PULL OVER. In a von Neumann or McCarthy machine, every routine for entering intersections, pulling out of parking places, changing lanes and driving along must be modified to call the new unit at the right moments. In a Newell machine, the SIREN production is simply added to the lineup with high priority; no existing production is touched, and it sits quietly until a siren comes along.
Two limitations counterbalance this:
- Not every task divides into autonomous parts. Sometimes subtasks need explicit coordination, with particular information sent to particular units at particular times; possible in production systems but distinctly awkward compared with centralised calls.
- Modularity is single-level. Productions are all on a par except for priority; you do not build high-level modules out of lower-level ones. LISP definitions are inherently hierarchical, and von Neumann programs often are.
Architecture as an Empirical Variable
| Architecture | Memory | Access | Control: who acts next | Conditionals | How you program it |
|---|---|---|---|---|---|
| Babbage's Analytical Engine (c. 1833) | Store of numbered columns; program on card strings | Store absolute; cards relative | Referee follows the card schedule | Occasional branches (move back n cards) | Punch a branched schedule on cards |
| Turing machine (1936) | One unbounded tape | Relative only | Each state nominates its successor | Every step conditional | Write a state-by-token chart (or code it for a universal machine) |
| Von Neumann machine (1940s) | One linear memory for data and program | Absolute and relative | Referee follows a prescheduled program; stack for returns | Occasional branches | Branched schedule with loops and subroutines |
| LISP / McCarthy machine (1959) | Many dynamic trees built to order | Name the tree, then a path of left/right turns | Functions invoked by other functions, goal first | IF chooses what gets evaluated | Define functions from six primitives, recursively |
| Production system / Newell machine (1960s) | One common linear workspace | By content (pattern match) | Productions self-activate; priority resolves conflicts | Every step conditional | Write condition → action rules |
Equally powerful (universal) architectures can be deeply and dramatically different, so from the perspective of AI, mental architecture itself becomes a new theoretical variable, to be investigated and spelled out by actual cognitive science research. AI makes no claim that the mind is a Turing machine or a BASIC program. Most AI programs are written in LISP for convenience, but the virtual machines they create are not LISP machines and "in principle could be anything". Some workers suspect the mind is organised rather like a production system, though far more elaborate; the point is only that the mind could have an architecture all its own.
What became of the architectures. Production systems had a commercial boom: Charles Forgy's OPS5 and its fast Rete matching algorithm (1979) underpinned expert systems such as DEC's XCON (R1), which configured VAX computer orders from about 1980, and Rete descendants still run business-rules engines today. As psychology they matured into cognitive architectures: Soar (Laird, Newell and Rosenbloom, 1987), the centrepiece of Newell's Unified Theories of Cognition (1990), and John Anderson's ACT-R, both still used to model human performance. The von Neumann design still rules general-purpose CPUs, but modern AI runs mainly on massively parallel accelerators (GPUs and TPUs) because neural networks are dominated by matrix multiplication; the cost of shuttling data between memory and processor (the "von Neumann bottleneck", named by John Backus in his 1977 Turing Award lecture) is now a central hardware concern. Haugeland's three access modes also reappear inside neural networks: the Neural Turing Machine (Graves and colleagues, 2014) combined content-based and location-based addressing of an external memory, and the attention mechanism of transformers retrieves information by matching a query against keys, a soft, learned version of Newell's content-based access.
For an enterprise architect, Haugeland's tour is a reminder that control structure is a design choice with consequences for change. A rules engine behaves like a Newell machine: adding a new regulatory check (the siren case) is a new rule with a priority, not a rewrite of every workflow, but conflict resolution, rule ordering and the lack of hierarchy become the risk, and large rulebases grow hard to audit. Orchestrated workflows behave like von Neumann or LISP machines: easy to reason about and test, but every new cross-cutting concern means editing every caller. Agentic LLM systems that pick tools from a shared context mix the two, and inherit both sets of trade-offs.
Real Machines
Haugeland now turns from what computers could be to what AI actually built in its first quarter century, and reads each phase as an experiment with a lesson. Cybernetics and machine translation failed because raw power and statistics ignored understanding; heuristic search (Newell, Shaw and Simon's Logic Theorist and GPS) tamed the combinatorial explosion but left the real intelligence in formulating problems; micro-worlds (Winograd's SHRDLU) showed that the world cannot be cut into independent fragments; commonsense stereotypes (frames, scripts) organised knowledge brilliantly yet stumbled on context; and the frame problem exposed how hard it is to keep beliefs about a changing situation up to date.
Pre-AI
Computers have always been impressive. The Analytical Engine would have been a wonder of the world, and the machines of the mid-1940s dazzled the press into phrases like "superhuman brain," "custom-built genius" and even "robot Einstein."
- ENIAC: front panel a hundred feet long, 18,000 vacuum tubes, thirty tons, two and a half years to build, just under half a million dollars.
- ENIAC's speed: thirty-eight nine-digit divisions per second, the speed king of 1946; it reportedly solved in two hours a problem that would have taken a skilled person a hundred years.
- ENIAC's memory: twenty "words" of ten decimal digits, plus separate storage for three hundred instructions.
- The Differential Analyzer of the same period weighed one hundred tons, with a million feet of wire; a 760,000-part gear-and-ratchet machine nicknamed "Bessie" (the Harvard Mark I), "would have made Babbage weep for joy."
- The Cray-1 (1985 "supercomputer"): about a million times faster than ENIAC with a million times the memory (four million 64-bit words, 20 to 100 million multiplications a second), for about six times the price in constant dollars (around $15 million).
- Net gain: performance per cost improved about 100,000 to 1; had air travel kept pace, we would hop between cities in minutes for less than a dollar.
- Size. An adult brain weighs about three pounds and contains roughly ten billion neurons (Haugeland's figure), with perhaps ten times as many support cells.
- Connections. Each neuron averages about a thousand interconnections, giving ten trillion per brain ("plus or minus a few trillion, but who's counting?").
- Two kinds of storage. A neuron cannot do 64-bit floating-point arithmetic, but with a thousand connections it "stores" far more than 64 bits: relatively permanent information implicit in which neurons connect to which, and transitory information in the cell's changing state. Computers have much explicit transitory memory but exceedingly little information implicit in their structure.
- Serial versus parallel. Contemporary computers are serial or nearly so; brains work massively in parallel, with millions or billions of processes at once. Semiconductors are about a million times faster than neurons, and speed trades off against parallelism.
- Verdict. The brain is many orders of magnitude more complex than any artifact, which is daunting but also reassuring: current programs' limits may not be fundamental, only a sign that more raw power is needed.
The numbers since 1985. Current counts give the human brain about 86 billion neurons (see Chapter 3) and a glia-to-neuron ratio closer to one to one than ten to one; synapse estimates run to around 100 trillion or more, ten times Haugeland's figure. Machines have moved further. The leading supercomputers of 2025, such as El Capitan at Lawrence Livermore, exceed an exaflop (10^18 operations per second) on the standard benchmark, billions of times a Cray-1's peak. Haugeland's guess that "more raw power" might matter proved prophetic in a way he did not intend: the decisive gains of the 2010s came from massively parallel GPUs training neural networks whose knowledge is stored implicitly in connection weights, exactly the kind of structural storage his box says computers lacked. Frontier models have hundreds of billions to trillions of parameters, still fewer than the brain's synapses.
Scale alone is not enough. Two false starts of the 1950s show why.
False start one: cybernetics
Norbert Wiener (1948) named the theory of systems and control "cybernetics". A system S meant to produce output O will have some error; there are two ways to reduce it: (1) build S better (more accurate parts, careful calibration), or (2) build S to measure its own output, detect errors and compensate automatically. In fancy systems the second is far more effective. The theory of such self-regulating systems is cybernetics, and its principle is negative feedback.
- Feedback. Take part or a consequence of the output (e.g. temperature) and use it to modify the input (heat transfer through walls and from equipment), as when a thermostat switches equipment on and off.
- Positive feedback exaggerates output changes and drives the value to an extreme (a thermostat wired backwards; switches that "click" to the nearest stop).
- Negative feedback cancels output changes and holds the value at an intermediate point; it is the more important concept.
- Three balls. A ball on a flat surface is in neutral equilibrium (no feedback); one in a valley is in stable equilibrium (negative feedback pulls it back); one on a hilltop is in unstable equilibrium (positive feedback rolls it off).
- Cruise control. Uphill, the regulator adds gas; downhill, less. Add too much too fast and the car surges, overcorrects and oscillates: another kind of instability, which is why the exact mathematics matters.
Negative feedback is "seductively analogous to purpose." If I wanted a constant temperature I would start the furnace when the room cooled, like a thermostat; when insects win the battle for your garden you add insecticide, which looks like compensation. From there it is a short step to cybernetics as the science of all human action: people are more complicated than thermostats, but "the essence of life is goal-seeking."
Except it isn't. Cybernetics as science is a mathematical theory of quantitative input and output displacements, related by differential equations in time; remove the equations and "little remains but handwaving and amusing parallels". A gardener does not act like an "insectostat": she notices damage, diagnoses the pest, looks up safe techniques, checks supplies, makes a plan and follows it. Perceptions, inferences, knowledge and plans are not quantitative magnitudes, so explaining thought cybernetically is as hopeless as explaining it by conditioned reflexes or Hume's association of ideas.
False start two: machine translation
The idea gained currency in 1949 through a memorandum by mathematician Warren Weaver and was pursued in the United States, England and Russia for at least a dozen years. The problem was split into grammar (declensions, conjugations, word order) and vocabulary (look up an equivalent for each word in a mechanical dictionary). Vocabulary was the trap: ambiguities rarely match across languages.
| English "suit" sense | German word | What else the German word means |
|---|---|---|
| Lawsuit | Klage | A lament; nothing about cards or clothes |
| Suit of cards | Farbe | The normal word for colour and paint |
| Man's suit | Anzug | A kind of approach or beginning |
| Verdict | No German "equivalent" for "suit" | Translation needs to know which sense is meant |
Weaver proposed a statistical fix: look at the N nearest words, so "card" or "club" favours Farbe and "judge" or "case" favours Klage. Haugeland's reply: tailors and lawyers have business cards and clubs, and anyone can judge the colour of a suit in a display case. Could a subtler "statistical semantics" (Weaver's phrase) win? "Not a chance."
The most amazing thing about early MT is the lack of effort to understand the text. Haugeland finds three causes:
- Wartime cryptography. Huge progress in statistical codebreaking; Weaver recounts a message encoded from Turkish decoded back into Turkish by someone who knew no Turkish (and thought he had failed). So why not treat Chinese as English encoded in "Chinese code"?
- Formal notations. Mathematicians knew many equivalent notations for formal languages that can be intertranslated algorithmically, and natural languages seemed formal deep down, only cluttered.
- Information theory. Weaver, with Claude Shannon, pioneered statistical information theory, concerned with reliable transmission through noisy channels, notoriously without attention to meaning. An exact codebook looked like a perfect channel, natural language like a noisy one: hence approximation and statistics, "all very hard-headed, modern, and scientific".
In 1951 Yehoshua Bar-Hillel became the first person paid to work on machine translation; nine years later (1960) he was the first to identify the fatal flaw and abandon it. His test sentence: "The box was in the pen." Considering only pen = writing instrument versus pen = child's playpen, any English speaker instantly picks playpen, by common sense: if one object is in another the latter must be larger, and fountain pens are much smaller than boxes. Why not encode such facts? Bar-Hillel: that would require "not only a dictionary but also a universal encyclopedia," which is "utterly chimerical."
A "mere" translator is impossible: any system capable of genuine, accurate translation must know what people know and apply it sensibly, so mechanical translation presupposes full-fledged Artificial Intelligence. Whether AI is "utterly chimerical" remains to be seen, Haugeland adds, but it deserves further discussion.
"Not a chance" aged badly. Haugeland's verdict on statistical methods is the book's most conspicuous miss. IBM's Candide project (Peter Brown and colleagues, around 1990) revived statistical machine translation from aligned parliamentary texts; Google Translate launched on statistical methods in 2006 and switched to neural machine translation in 2016; and the transformer itself was introduced in 2017 in a translation paper ("Attention Is All You Need"). These systems do resolve "suit" and "pen" from context far better than any N-nearest-words scheme, because they learn rich contextual representations from billions of sentences. Bar-Hillel's half of the argument survives in altered form: modern systems succeed partly by absorbing something like the "universal encyclopedia" from text, so the debate has shifted to whether that statistical absorption amounts to knowing.
Heuristic Search
Artificial Intelligence, Haugeland writes, was "conceived at Carnegie Tech in the autumn of 1955, quickened by Christmas, and delivered on Johnniac in the spring," with a stunning June debut at the Dartmouth Summer Research Project on Artificial Intelligence (McCarthy's conference, which gave the field its name). The parents were Allen Newell, Cliff Shaw and Herbert Simon (NS&S), whose Logic Theorist "stole the show". What set them apart from cybernetics and MT was an explicit focus on thinking: intelligence is the ability to solve problems, and solving problems is finding solutions by heuristically guided search. (A note adds that they recognised search as the common thread only retrospectively; the reliance on heuristics was there from the start.) Puzzles came first, but ordinary activity casts easily as a quest: "find" a way to start the car, "look for" something to say.
Generate and test
Every search has an object (what is sought) and a scope (where it is sought), and a designer must make both explicit:
- The tester. A system cannot seek what it cannot recognise, so it needs an executable test for success, and that test defines what the system is really seeking.
- The generator. It needs a procedure that produces candidates by slogging methodically through the possibilities, neither searching the yard for keys lost in the house nor only one room repeatedly; the generator defines the effective search space.
- The cycle. Generator proposes, tester checks; on success stop, otherwise go round again until the space is exhausted.
The combinatorial explosion
Chess defines a tree-shaped search space: nodes are positions, branches legal moves, leaves checkmates and draws. Minimaxing turns the end-tests into an optimal current move: work backwards, assume the opponent never errs, prune every branch leading to a loss or tie. This "rosy theoretical analysis" is "utterly and absurdly infeasible."
- 30 to 35 legal options in a typical chess position.
- About 1,000 possibilities to look one full move ahead (each side plays once).
- A million for two moves, a billion for three, a trillion for four.
- 10^120 combinations for a forty-move game, more than the number of distinct quantum states of all subatomic particles in the history of the known universe. Haugeland "boldly predicts" that no computer will ever play perfect chess by exhaustive search.
This is the combinatorial explosion, and it haunts almost every interesting search; controlling or circumventing it has been a central concern of AI from its inception. Search must therefore be selective: partial and risky. The insight of NS&S is that selection need not be random: problem-solving search follows heuristic guidelines, and they even suggest that the degree of improvement over chance measures a system's intelligence. Applying heuristics is what it means to think about a hard problem, and designing an intelligent machine reduces to finding powerful heuristics.
GPS and means-ends analysis
After the Logic Theorist, NS&S built the General Problem Solver (GPS) around means-ends analysis. Abstractly, every problem has (1) a specified initial state (premises, position, data), (2) a specified goal state or states, and (3) a set of operators transforming states; the task is to find an operator sequence from initial to goal. GPS is given two kinds of heuristic: (1) procedures to detect salient differences between states (and rate their importance), and (2) rules of thumb about which operators typically reduce which differences.
- Side effects. If an operator makes the problem harder, GPS can back up to an earlier state and work on a different difference.
- Preconditions. If an operator does not quite apply, GPS sets a subgoal of reaching an intermediate state that meets its conditions, so it can back away from dead ends and plan ahead a little.
- Why "means-ends" and "general". Operators are means, goals and subgoals ends; the means-ends apparatus is separate from any particular heuristics and operators, which can be swapped per problem. The intuition: reflecting on what you need and how to get it is fundamental to all thoughtful problem solving.
NS&S meant GPS to solve problems the way people do and treated program design as a new way of formulating psychological theories. They tested it by protocol analysis: an undergraduate solves a problem thinking aloud, GPS solves it in "trace mode", and the transcripts are compared. The match was imperfect but remarkable for its time.
A student inexperienced in logic and GPS both had to transform (R ⊃ ¬P) ∧ (¬R ⊃ Q) into ¬(¬Q ∧ P) using twelve given rules (from Newell and Simon, 1961). The student considers rule 8 and rejects it (it would eliminate Q or P), applies rule 6 to both sides, sees he "should have used rule 6 on only the left hand side", redoes it, finds rule 7 blocked by the horseshoe, and ends: "(Sigh.) I'm just sort of lost for a second." GPS's trace runs Goal 1: transform L1 into L0; Goal 2: delete R; apply R8, rejects that, tries deleting the other half, rejects again, then pursues R7 via R6 on the left and on the right, undoes its own step, and lands on "we've been here before". Both wander into the same temporary dead end by similar routes.
A side lesson about computers and logic. Fictional computers are frightfully "logical", as if inference were as trivial for them as addition for a calculator. Yet GPS, on a big machine, "barely manages to bumble through elementary logic at novice level." The point is not that GPS is primitive but that general problem solving is a far cry from calculation, even when the reasoning is logical; machines that think like us won't find logic especially trivial either.
A dream come false
GPS's generality rested on two unfulfilled assumptions.
- All problems are alike under the skin. Once formulated, every problem should yield to the same general techniques, with only the conditions, goals and a few hints varying. In fact powerful general methods proved hard to find, while special-purpose methods paid off handsomely. By the mid-sixties this produced "semantic programs", rich in domain-specific data structures, heuristics and shortcuts (semantic in the strong sense of carrying detailed factual information about a domain). They performed better but became brittle specialists, "idiot savants" brilliant at one task and helplessly inept at everything else. A way was needed to get detailed knowledge without overspecialisation.
- Formulating a problem is the small job. This assumption undermines heuristic search in general: experimenters optimised the formulation themselves and gave the machine only the solving. But choosing a good search space and heuristics is both difficult and important, while the search itself, though tedious, is routine; much, perhaps most, of the intelligence in problem solving lies in that "preliminary" stage. (Haugeland sets aside yet another aspect of intelligence: knowing when there is a problem.)
The mutilated checkerboard. Thirty-two dominoes, each covering two adjacent squares, exactly cover a 64-square board. Remove two diagonally opposite corners: can thirty-one dominoes cover the remaining sixty-two? Formulation one: search all distributions of thirty-one dominoes for one that leaves the corners bare, "a long haul", never finished in our lifetimes. Formulation two: opposite corners share a colour, so the board has 30 squares of one colour and 32 of the other; each domino covers one of each, so any coverable board has 31 and 31. The search space has two members and the answer (no) is immediate. Both formulations are easy to program, and with the second "there's nothing left for the machine to do." The real challenge is getting from the informal English statement to a good formulation.
Even so, explicit selection heuristics to tame the combinatorial explosion were a major intellectual milestone, perhaps the decisive one in launching AI, and a conceptual mainstay ever since. The Logic Theorist was not algorithmic under its interpretation: it pursued a symbolically specified goal by sensible symbolic explorations guided by symbolically coded knowledge, making it the first human artifact plausibly describable as solving problems by thinking about them.
Search after GOFAI. The Logic Theorist proved 38 of the first 52 theorems of chapter 2 of Whitehead and Russell's Principia Mathematica, one more elegantly than the original. Heuristic search remains a pillar of AI: A* (1968), Monte Carlo tree search and the AlphaGo family (Go, 2016) all combine a generator, an evaluator and heuristic pruning, the last with learned heuristics (on chess engines see Chapter 2). The formulation problem Haugeland highlights is precisely where large language models changed the picture: they are good at turning an informal English problem into a formal representation (code, equations, Lean statements) that classical solvers can then search, which is how the neurosymbolic systems discussed in Chapter 3 work. Newell and Simon's later book Human Problem Solving (1972) developed protocol analysis into a major method of cognitive psychology.
Micro-Worlds
A micro-world is a contrived artificial domain in which the possible objects, properties and events are all narrowly and explicitly defined in advance. Chess is one: unlike a medieval warlord, a chess strategist never worries about plague, excommunication, rumours in the ranks or Cossacks on the horizon. Monopoly is "a crisp micro-caricature of the real estate business."
By 1970 micro-worlds were all the rage, especially MIT's blocks world: a few imaginary blocks of simple shapes with perfectly flat surfaces on a flat table, sometimes coloured or shadowed, but of no material, mass or texture; they cannot get lost, chewed by the dog, traded for water pistols or burnt wistfully on a cold winter night.
- The motive is generality. The MT collapse showed that language users must know what they are talking about. Heuristic problem solvers carry "predigested expertise" but no grasp of their subject: a program could solve missionaries-and-cannibals puzzles and "still not know a cannibal from a bowling ball or where to pass the ketchup if a missionary has a heart attack."
- Thorough knowledge. A micro-world can be represented completely, a first approximation to common sense; the blocks world at least mimics basic spatial properties.
- Not task-specific. There were blocks-world studies of perception, planning, action, talking and learning, all in one world, so they could presumably be integrated into one system.
- The physics analogy. Simplicity is a temporary expedient, like early physicists ignoring friction and deformation to find universal laws of motion.
SHRDLU
The best known and most impressive blocks-world program is Terry Winograd's (1971) SHRDLU, a simulated robot built mainly for conversation in English. He handles complex noun phrases and ambiguous pronouns as long as the talk is about the blocks on his table, resolves unclear sentences from earlier dialogue or the current arrangement, carries out commands on screen, performs prerequisite actions unasked, explains why he did things, and learns new words by definition.
- "Pick up a big red block." OK. (To do it, he first cleared off a green block by finding space for it.)
- "Grasp the pyramid." I DON'T KNOW WHICH PYRAMID YOU MEAN. (There are three.)
- "Find a block which is taller than the one you are holding and put it into the box." BY "IT", I ASSUME YOU MEAN THE BLOCK WHICH IS TALLER THAN THE ONE I AM HOLDING. OK. (He must resolve "one" and "it", deduce what he is holding, and put down the red block to take the taller blue one.)
- "What does the box contain?" THE BLUE PYRAMID AND THE BLUE BLOCK.
- "What is the pyramid supported by?" THE BOX. (Now "the pyramid" means the one just mentioned.)
- "Does a steeple..." SORRY, I DON'T KNOW THE WORD "STEEPLE". Told "A 'steeple' is a stack which contains two green cubes and a pyramid," he answers I UNDERSTAND; asked "Are there any steeples now?", NO; told "Build one," OK, deducing a construction plan from the definition.
SHRDLU is captivating, so Haugeland lists his obvious deficiencies first:
- He never initiates anything. He has no motivational structure beyond goals put to him from outside.
- No sense of conversational coherence. Gratuitous jumps from "Pick up a big red block" to "Grasp the pyramid" do not bother him; a person might tell you to "make up your damn mind."
- No real perception or action. The hard problems lie at the interface between symbolic cognition and nonsymbolic objects and events, but SHRDLU's world is already symbolic, a description elsewhere in the computer he reads and writes with "handy little read/write routines". (His action routines do refuse to put two blocks in one place or anything on a pyramid, and he reports failure "as if he had actually been thwarted by gravity".)
The fundamental limitation is the micro-world itself
SHRDLU is glib only because his domain has been stripped of anything requiring genuine wit. Ignoring the tangles of everyday life while seeking a theory of common sense is not like ignoring friction while seeking laws of motion; it is throwing out the baby with the bathwater. A frictionless wheel approximates a real one because the deviations are small and theoretically localised; the blocks world "approximates" a playroom more as a paper plane approximates a duck.
- "Trade you a squirtgun for a big red block." SORRY, I DON'T KNOW THE WORD "TRADE".
- "A 'trade' is a free exchange of property." SORRY, I DON'T KNOW THE WORD "FREE".
- "A 'free' act is done willingly, without being forced." SORRY, I DON'T KNOW THE WORD "ACT".
- "'Acts' are what people do on purpose, and not by accident." SORRY, I DON'T KNOW THE WORD "PEOPLE".
- "Sorry, I thought you were smarter than you are." SORRY, I DON'T KNOW THE WORD "SORRY".
Nor does he know property, force, accident or smart, or the deeper notions of wanting, having and agreeing without which trading makes no sense.
The problem is deeper than vocabulary: trading concepts would be unintelligible in the blocks world even if added to the dictionary, so the domain itself must be enriched. Why not a "blocks-and-trading" micro-world? Because it can't be done: trading cannot be separated from the rest of everyday life; there are no plausible, nonarbitrary boundaries on what might be relevant. Imagine SHRDLU answering normally: "I ALREADY HAVE A WATER PISTOL, BUT I'LL GIVE YOU TWO BLOCKS AND A PYRAMID FOR YOUR SLIMY OLD FROG." That childish reply presupposes:
- A second squirtgun is worth less than the first, so he knows how to play with them and can visualise shooting two-handed.
- He expects us to know that (and know that he knows), so we see he is giving a reason and declining.
- We want a block badly enough to part with something important, and blocks, unlike pistols, are valuable in quantity.
- He softens our reluctance by demeaning the frog (assuming slimy old things are less valuable), and is prepared to haggle.
These "but scratch the surface": a simple negotiation can involve pretty much any aspect of common sense. Unbounded potential relevance is antithetical to micro-worlds, so a trading micro-world is impossible.
What SHRDLU gets away with: memory. He remembers everything that happens (until restarted, then nothing), feasible only because his world is tiny and sessions short; a real system needs selective memory and a sense of what is worth remembering, of which he "hasn't a clue". Conversely he remembers nothing of the present situation, simply re-checking the tidy world whenever he needs to, so he is spared memory updating, including how a change in one belief entails changes in others.
What SHRDLU gets away with: situation. He has no sense of situation, because the blocks world has one thin topic. Real behaviour depends on circumstance: "Block the big red pick-up" means something different in a rowdy singles bar, an auto repair shop, football practice or a spy adventure. Even with all the concepts, he would not know how they relate or which make sense where, so he would not understand them; and by the same token he does not understand the words he already has, a fact masked by the blocks world.
Some real domains are micro-worlds, chess among them. Expert systems exploit the discovery that many domains can be treated as micro-worlds for certain decisions, given four conditions:
- Decisions depend entirely on a well-defined, not too large set of variables.
- Their values are known or discoverable and can be expressed articulately.
- How decisions depend on them is known and "computable", at least approximately.
- The interrelations are complex enough to make the project worthwhile.
These rule out most of ordinary life, but suit specialised domains, including (1) diagnosis and chemotherapy for certain infectious diseases, (2) geological analysis of drill-rig bore samples for strata and petroleum, and (3) optimising integrated-circuit layouts under heat, delay and interference constraints. Expert systems are practically important but not GOFAI or cognitive science: built for a limited job, they are immune to micro-worlds' theoretical flaws and contribute "little or nothing" to understanding common sense.
Haugeland's conclusion is "rather negative": as a tool for exploring general intelligence and common sense, micro-worlds are thoroughly misguided. Simplifying is reasonable, but the broad interconnectedness and situation dependence of common sense are central, not incidental; ignoring them evades the fundamental problems. SHRDLU's successes are gimmicks, demonstrations that on a contrived stage "with clever baffles and mirrors, you can give a startling impression of something that isn't there": SHRDLU can no more speak English than Houdini can saw his assistant in half. This is 20-20 hindsight, with no criticism of Winograd: as MT's failure established that language requires understanding, the micro-worlds effort showed that the world cannot be decomposed into independent fragments, a point long made by Husserl, Heidegger, Dewey and the later Wittgenstein, now established empirically.
The expert-system era and its end. Haugeland's three examples match famous systems: Stanford's MYCIN (1970s) for infections and antibiotics, geological systems such as SRI's PROSPECTOR (mineral exploration) and Schlumberger's Dipmeter Advisor (oil-well logs), and design tools for chips. The 1980s boom (DEC's XCON, corporate "knowledge engineering", Japan's Fifth Generation project from 1982) ended in the late 1980s: systems were brittle at their boundaries, costly to maintain, and stalled on the "knowledge acquisition bottleneck" of extracting rules from experts. His verdict that expert systems teach nothing about common sense matches how the field later judged them. Winograd himself turned critic: with Fernando Flores he wrote Understanding Computers and Cognition (1986), a Heidegger-inspired argument against the symbolic programme very close to this chapter's. Today's LLMs are the opposite of micro-world systems: trained on everything and specialised on nothing, they can banter about trading squirtguns for frogs but have no tidy world model to check, which brings back the reliability problems SHRDLU never faced.
Commonsense Stereotypes
Pious Pat: "We bought the boys apples, because they were so hungry." Cagier Chris: "We bought the boys apples, because they were so cheap." Identical until the last word, yet "they" means the boys in one and the apples in the other, and native speakers rarely notice the ambiguity. (A note credits a similar example to Yorick Wilks (1974); translating into French forces the choice, since they takes a different gender.) The commonsense reasoning is trivial (boys eat and apples don't; apples have prices and children don't), but for AI the hard questions remain: how would it "occur" to a system to reason that way, using these facts rather than countless others about tigers' stripes, knobs or the Queen of England? Calling prices relevant and queens irrelevant just renames the issue: how does a system determine what is relevant?
Common sense ranges fast and wide: almost any general knowledge can turn out pertinent in almost any conversation. Raw speed and size are not the issue (microsecond, megaword memories were commonplace); efficient access is: finding what you want without tedious search. Think of memory as a super filing system: storage is easy, retrieval years later is the challenge, solved by indexes, hierarchical categories and cross-references. In AI this is the knowledge access or knowledge representation problem.
By 1975, representing knowledge in linked stereotypes was "an idea whose time had come." Stereotypes are general descriptions, in a conventional format, of familiar objects, events and situations: like definitions but far more encyclopedic, and necessarily "painfully thorough and banal", because common sense must start from zero with articles on tying shoes and getting wet in the rain. Haugeland's neutral term covers what others call "noemata" (Husserl, 1913), "schemata" (Bartlett, 1932), "frames" (Minsky, 1974), "preference structures" (Wilks, 1974) and "scripts" (Schank, 1975).
Take Bar-Hillel's sentence again. A system has one stereotype each for "box" and "in" and two for "pen". The "in" stereotype specifies a relation in which the second object surrounds a cavity where the first is located, with a prominent prerequisite: the first object must be smaller than the cavity. Alerted, the system looks up the size specifications in the other three stereotypes and resolves the ambiguity at once. Fifteen years after Bar-Hillel, nearly everyone accepted his diagnosis and therapy, but not his despair.
| Gain | How it works | Example |
|---|---|---|
| 1. Never considering the irrelevant | Only stereotypes indexed by the sentence's words are consulted; tigers and knobs are never considered at all, not considered and dismissed | Four stereotypes settle the box in the pen |
| 2. Constraints | Some stereotypes (notably for predicates) impose conditions on what may combine with them | "Is hungry" requires an animate subject, recoverable from "boy" but not "apple" |
| 3. Default assignments | Features carry modal qualifications; "typical" values are provisional assumptions that can be overridden | Boxes are typically a few inches to a few feet, but Agent Orange's "tiny box" of films can be in Raoul's pen |
| 4. Cross-referencing | Shared facts are stored once in a more general stereotype and inherited, often in a hierarchy | Frogs, toads and salamanders inherit amphibian facts; frogs share vertebrate facts with eels and elephants |
Modal qualifiers go beyond "typical": frogs are always four-legged (unless defective), usually brown or green (except poisonous species, which tend to bright colours), occasionally pets, never ferocious. Mention a frog and I will assume brown or green but stand ready to learn otherwise (and fear poison if I do); I will be highly confident it is a mild-mannered quadruped, and unsurprised if it is a pet.
Indentation shows subordination, parenthesised words are modal qualifiers, capitals point to other stereotypes (a real AI stereotype has far more structure).
- Subclass of (always) DOG; specifies (always) BREED.
- Coat (always) = FUR, HAIR; covering (almost always) entire SURFACE except EYES, NOSE, PAWS; colour (usually) BLOND or BLACK; care (needed) BRUSHING, (ideally) twice a WEEK, on pain of (inevitably) SHEDDING, MATTING, STINKING.
- Tail (always) TOP REAR; type (almost always) STUBBY; state (usually) WAGGING OBSEQUIOUSLY.
- Adult size (almost always) MEDIUM for a DOG, SMALL for a SPANIEL; typical 15 inches high, 9 wide, 30 long, 25 pounds.
- Habits: good (typical) LOYAL and PLAYFUL; bad (too often) BARKS and JUMPS, (sometimes) hard to HOUSEBREAK, (whenever possible) STEALS FOOD.
- Examples: (known) AUNT MAUD's BOWSER III, the ROBINSONS' GOLDEN LADY, the BLACK one who CHASES our CAT; (heard of) BOWSER JUNIOR and BOWSER WOWSER; (metaphor) AUNT MAUD's HUSBAND, CALVIN.
- Miscellaneous: (definitely) POPULAR; (presumably) GOOD with CHILDREN; (allegedly) PUPPY MILLS are WRECKING the BREED.
Unlike micro-worlds, stereotypes do not evade the tangles of daily life; they try to duplicate the organisation of common sense "in all its irregular glory", a profound idea with formidable theoretical power. But three stubborn difficulties remain.
| Problem | The difficulty | Haugeland's examples |
|---|---|---|
| Selection | How are the needed stereotypes picked out without explicit word cues? | "Where is Johnny's toy box?" "Try his pen" (no "in", so no size check; and what is "trying" a pen?). "The brown rectangular object labelled INDIA INK, 48 BOTTLES, HANDLE WITH CARE was in the pen" (no "box", and no typical size for brown rectangular objects) |
| No topic | The relevant knowledge has no natural topic to be filed under | "I left my raincoat in the bathtub, because it was still wet." "It" is the raincoat because a wet raincoat is an intelligible reason and a wet bathtub is not, but that reason belongs to no stereotype for raincoats, bathtubs or wetness. (Charniak's goat: "Jack, did you do that?" refers to no word at all) |
| Tunnel vision | Local constraint checks cannot capture an overall appreciation of the situation | "When Daddy drove up, the boys quit playing cops and robbers. They put away their guns and ran out back to the car" versus "When the cops drove up, the boys quit their robbery attempt...": tidying or concealing? "Ran" as scampered or fled? |
The larger issue is how knowledge is brought to bear in context. Stereotypes work well when the context is a single predigested, explicitly identified topic with local constraints; once the text opens up, "all that elaborate machinery lurches and stumbles". It is too soon for a final verdict, but the showing so far is "disquietingly reminiscent of micro-worlds": clever staging giving a startling impression of something that isn't there.
Common sense, from Cyc to LLMs. The universal encyclopedia Bar-Hillel called chimerical was attempted head-on: Doug Lenat's Cyc, begun in 1984, spent decades hand-encoding millions of commonsense assertions with inheritance and defaults, and never delivered the general understanding it aimed at. Stereotype ideas survive in object-oriented programming, ontologies and knowledge graphs (inheritance hierarchies with typed relations), and formal logics of defaults and non-monotonic reasoning (Reiter's default logic and McCarthy's circumscription, both 1980). Haugeland's pronoun puzzles became a benchmark: Hector Levesque's Winograd Schema Challenge (2011-2012), named after a 1972 example of Winograd's, used minimal pairs like Pat and Chris's ("the trophy doesn't fit in the suitcase because it is too big / small"). Large language models reached near-human accuracy on it and its larger successor WinoGrande by the early 2020s, and a 2023 paper by Kocijan, Davis, Lukasiewicz, Marcus and Morgenstern was titled "The Defeat of the Winograd Schema Challenge", while warning that passing it showed less commonsense understanding than its designers had hoped, partly because models exploit statistical cues. Nobody filed the raincoat-in-the-bathtub reason under a topic; LLMs store such knowledge distributed across weights, which dissolves the no-topic problem as Haugeland posed it and replaces it with the problem of knowing what the model knows.
The Frame Problem
Papa Box rests in the middle of the cottage floor with Baby Box perched on his back; Mama Box sits on a chair by the door. Goldiwires, "bent on problem solving", pushes Papa across the room to the opposite wall. IQ test: how many Boxes are there now? Three, obviously, but how does the robot "see" that? Then: where is Papa? Mama? Baby? Goldy knows where Papa went (she pushed him) and may suppose Mama and Baby stayed put, since she never touched them. But moving Papa has a side effect: Baby comes along for the ride, though Mama does not.
Side effects are hard to anticipate because they follow incidentally, from details of the particular situation (Baby happens to be atop Papa), yet checking all conceivable consequences is out of the question (did opening the door break the window, let out the cats, or change the number of Boxes?).
The frame problem (first described and named by McCarthy and Hayes, 1969) is how to "notice" salient side effects without having to eliminate all the other possibilities that might conceivably have occurred had the facts been different. The difficulty is not deciding, for each consideration, whether it matters, but avoiding that decision in almost every instance.
The frame problem resembles the knowledge-access problem (both are about selectively ignoring almost everything), but differs in temporality. Knowledge access concerns generic, timeless information (pens stay thin); the frame problem concerns knowledge of the current situation, here and now, and keeping it up to date as events unfold. Knowledge must divide into perennial facts and temporary facts that must be actively maintained. That makes the frame problem harder in two ways:
- Interdependence. Updating requires moving around the knowledge base according to how entries depend on one another, another whole level of combinations.
- Situation-bound dependencies. The dependencies themselves depend on happenstance details: Baby rides along this time, maybe not next time, so update decisions cannot be worked out in advance. (A note adds that real situations compound this with continuous changes (the rat and mongoose darting about the dollhouse), simultaneous interacting events (Ajax spills the turpentine just as Comet hits the lantern) and cumulative effects (the last straw on old Dutch's back).)
Two schools
- The cheap test school. Scan the whole represented situation and rule most of it out at a glance. Goldy knows pushing changes location but not colour, size, shape or weight, since these are normally independent; with a prior classification of which event types affect which fact types, each fact can fly a flag (say, bright red for "affected by movements") and the rest are skipped. A variant assumes no action at a distance: flags mark objects in causal proximity (Baby touches Papa; Mama is across the room). Trouble: the real world isn't so compartmentalised. "Down in Philadelphia", Mama may well be affected if Papa is shoved against a wall, changes may include her colour and shape, and effects can ramify to the contents and velocity of a frying pan. Make the tests discriminating enough and they are no longer cheap.
- The sleeping dog school. Let things lie unless there is a positive reason not to, with no tests at all. Satisfying, but how does it occur to the system to worry about Baby's position but not Mama's or the number of Boxes? Two approaches have currency.
- Split temporary facts: (1) basic facts are all independent of one another; (2) nonbasic facts are all derivable from basic facts. Then there are no side effects among basic facts, and the frame problem is confined to derived ones. (Independence here means that any allowed change to one basic fact alters no other, not that every combination is allowed.)
- Chess as the model. The locations of the pieces and whose move it is are the only basic facts; who is winning, which pieces are threatened and which moves are good are derived.
- Split derived facts again: (1) conclusions already explicitly drawn and (2) conclusions still implicit. Implicit ones update themselves "at the very speed of logic"; explicit ones live in danger of having the rug pulled out. Discarding all conclusions at every update is wasteful; better to link each basic fact to the inferences that used it.
- Where it fails: which facts are basic? If Mama's and Papa's locations are basic, so is Baby's, and then Papa moves while Baby stays behind, suspended in thin air. Make positions basic relative to their supports, and Baby rides along, but what if Baby is tied to the doorknob with a rope, or Papa tips over? And the Boxes' cottage is nearly a micro-world; picking basic facts in an army field hospital would be far messier.
- The idea. Capture typical side effects in elaborate stereotypes for situations and their changes: "move something" says that anything supported comes along for the ride, unless roped to the doorknob, provided the support doesn't tip over, as long as there are no low bridges, and so on.
- Where it fails. The hedges need hedges (Baby glued to Papa, the doorknob glued to Papa), and the stereotypes get hairy, so the frame problem is not resolved but relocated into finding one's way around the stereotypes.
- The JERK problem. AI "intelligence" is oddly inconstant: a system cruises along sensibly, then "out of the blue" bumbles ridiculously, as if it had no sense at all.
- Two reactions. Gleeful opponents declare the impostor unmasked, "just a clockwork orange after all"; embarrassed designers patch the case with another clause: "nothing fundamental, just a little trip up." It keeps happening.
- The pessimistic reading. AI lacks people's graceful "horse sense"; adding clauses is like greasing a rusty bearing, hiding the problem a little longer, so graceful and jerky differ in kind.
- The caution. "Always beware the swift dismissal." A slow movie lurches from pose to pose but blends into fluid motion when sped up; a photograph's grains vanish at enough resolution. Perhaps our easy good sense rests on sufficiently fine-grained rules and stereotypes, as smooth vision rests on a grainy retina (a point Haugeland credits to Dennett, 1978).
Haugeland presses the worry through typicality. Horses have exactly four legs, a matched set in standard places, a specifically equine skeleton, and few ways to be ridden. Living rooms have walls, windows, soft chairs and rugs, but any number of windows anywhere, furniture that varies enormously and often doesn't match, and countless ways to hold a loud party or quiet conversation. Situations vary even more than rooms, especially in the middle-level details that determine side effects. So situation stereotypes face a dilemma:
Omit the middle-level detail (mention furniture and windows but not what or where they are), and the frame problem is untouched, because side effects depend on exactly those specifics: is Baby on Papa this time, or on the table next to him?
Leave open slots ("variables") for the specifics, and something must fill and update the slots as things change: when Papa moves, how is the living-room stereotype readjusted? That is the frame problem all over again, now as the problem of applying stereotypes concretely, in real time.
AI has nonetheless grown enormously in its quarter century: problems that could not even be posed in the late fifties are now topics of vigorous research; by disciplining theory, the computational model also liberates it; even the failures are a wealth of unprecedented empirical knowledge; and each year brings new capabilities. Can we extrapolate from present trends to ultimate success? That needs a clearer idea of the goal, the business of Chapter 6.
The frame problem since 1985. It split in two. The logical frame problem (how to state the effects of actions without endless axioms saying what does not change) received workable technical solutions: Raymond Reiter's successor-state axioms (1991), the event calculus, and Murray Shanahan's book Solving the Frame Problem (1997), after detours such as the "Yale shooting problem" (Hanks and McDermott, 1986-87), which showed naive default reasoning going wrong. The philosophers' frame problem, Haugeland's version about relevance, sharpened by Daniel Dennett's robots that fail to notice a bomb on their wagon ("Cognitive Wheels", 1984) and by Jerry Fodor, has no agreed solution. Deep learning sidesteps rather than solves it: networks learn implicit, graded relevance from data, and LLMs track many everyday side effects (they will tell you the baby box moved). But benchmarks built on the blocks world itself, such as PlanBench (Valmeekam, Kambhampati and colleagues, 2023), found LLMs poor at classical planning and state tracking; reasoning models from 2024 on (such as OpenAI's o1) did far better on standard problems yet degraded on disguised versions of the same puzzles. Haugeland's JERK box describes the result precisely: smooth competence punctuated by inexplicable blunders, the pattern Ethan Mollick and colleagues called AI's "jagged frontier" (2023).
Goldiwires in the bank. A customer moves house. The address field updates cleanly (a basic fact). Which derived facts now need rechecking: the branch allocation, the tax residency flag, the postcode-based fraud score, the mortgage property's occupancy status, the marketing consent tied to a regional campaign, a pending card delivery? None is "touched" by the address change, just as Goldy never touched Baby, yet each may ride along. A cheap-test design ("address changes affect only contact data") will miss some; a sleeping-dog design needs explicit dependency links from the address to every inference that used it. The thought experiment shows that the frame problem is not exotic robotics: it is every enterprise's data-lineage problem, and it gets worse exactly as Haugeland predicts, when which facts depend on which varies by customer situation.
Haugeland's sequence is a usable checklist for AI in banking. Expert systems succeed where his four conditions hold (credit-policy rules, sanctions-list matching, eligibility checks): keep them as auditable rules. Micro-world demos are the biggest risk in AI procurement: a copilot that dazzles on a curated scenario may be SHRDLU on a stage; test it on open, messy, situation-dependent cases. LLM assistants have the reverse profile: broad common sense, jerky reliability, so the controls that matter are grounding in authoritative data, explicit state and dependency tracking outside the model, and monitoring for the "we've been here before" loops and sudden blunders the JERK box describes.
Real People
Having looked forward from AI's history and achievements, Haugeland now looks backward from the goal, taking real people as the measure and scouting for omissions: phenomena that matter to intelligence but have not been (perhaps cannot be) absorbed into GOFAI. He examines four: pragmatic sense, mental imagery, feelings and ego involvement, and for each asks whether a segregation strategy could wall it off from cognition. He offers no disproof of AI, rejects the "debunkers", reviews the fate of Simon and Newell's 1957 predictions, and ends as a skeptic: GOFAI is neither impossible nor inevitable, and after thirty years the hard questions remain open.
Pragmatic Sense
The chapter returns to the unsolved problem of making sense left over from Chapter 3, where interpretation, and therefore computation, was shown to depend on it. For symbol systems, telling the truth is the exemplary way of making sense. But nonsymbolic systems can make sense too, by acting intelligently.
Mice have a modest worldly savvy but utter precious few symbols, and whatever goes on in their brains, we credit them with wits because of their crafty little deeds. Wee Juan smells cornbread on the sideboard mingled with the scent of Felix the cat below; after a moment's hesitation he retreats into the wall, scampers up the back route and emerges unseen behind the dishrack. We explain this by ascribing beliefs (about cats, cornbread and the kitchen), preferences (eating over being eaten) and rationality: he weighs his options and makes a canny move.
Ascribing mental states from behaviour works much like interpreting alien symbols: the result must make pragmatic sense. Three ascriptions would be absurd:
- The belief, as he glances trembling at Felix, that he beholds a stately sycamore.
- The intent, as he scurries up the studs, of lying still and drifting off to sleep.
- The considered decision, based on sniffs and shown in scrambling, to run for Governor of Utah.
Each is absurd because there is no sensible relation between the ascribed state and the perceivable situation or the action performed. Truth reappears here as a sensible relation between perception and what is perceived, joined by a counterpart, success, linking intent to accomplishment.
- The principle. Ascribe beliefs, goals and faculties so as to maximise a system's overall manifest competence.
- Competence defined. The ability to achieve goals by making rational decisions in the light of knowledge and perception.
- The method. "Rationalise" the system's behaviour relative to its circumstances by interpolating just the right mental structure to do the job.
People both talk and act, so interpretation and ascription must be coordinated. If Blanche says "Watch out! That chair is hot" and then sits on it herself, something is amiss: perhaps we misread her, perhaps she likes a heated seat, perhaps she lied to save her place. Either way word and deed must jibe as the work of a single mind; symbolic and pragmatic sense together define a more general condition of meaningful coherence.
Talking is acting
Could AI concentrate on language and ignore the sense in actions, since Turing's test makes English fluency sufficient for intelligence? No, because talking is itself a kind of action. Speech acts are at once symbolic expressions and things we do to further our ends, so pragmatic sense is built into linguistic competence. If you want help in the library, ask; if your neighbour's dog is loose, tell him; if your plans must stay secret, keep quiet. Saying "I accept your offer" both expresses consent and pursues my goals, and also enters the institution of property exchange, with its own constraints: knowingly promising what I cannot deliver makes my agreement as defective as my morals. No system negligent of such matters could truly speak the language.
- Mousetrap psychology. The trap wants to kill mice and believes it can whenever one nibbles the bait; when it feels its trigger move, it comes to believe a mouse is nibbling, and it acts. Its beliefs are mostly true, its perceptions reliable, its actions consistent and successful. The schema is satisfied, yet the ascription is silly and gratuitous. So the ascription schema is not sufficient.
- Supertrap. When a mouse is soaked in gasoline, he strikes a match; under the shelf, he topples a dictionary; in a blender, he trips the switch; on the bait, he snaps shut. A common malevolent thread is generalisable by, and only by, ascribing a persistent goal: dead mice. Without malice aforethought, each lethal, exquisitely timed gesture would be a grisly, incredible coincidence.
- Hanging together. Mental ascriptions are like revolutionary warriors: they hang together or they hang separately. Each should figure in many rationalisations in combination with others: the beliefs about gasoline and blenders should also explain how Supertrap starts a bonfire or makes a milkshake, and subtler combinations explain why he spares cats and barn owls that nibble or reek of gas. Haugeland calls this tangled conspiracy semantic intrigue; it is what ascriptions need, beyond the schema, to be nongratuitous.
- Errors need a pattern. There are no flaws in chaos: mistakes are identifiable only against a largely reliable pattern. Jiggle a mousetrap with a feather and it snaps; did it try to kill "that mouse", believe "crush a feather, kill a mouse", or take target practice? None makes sense. With Supertrap, evidence that he expects sneaky mice in disguise and knows where they get feathers could justify ascribing a particular error.
Conversational implicature
More interesting is how pragmatic sense "piggybacks" on symbolic content and conveys something extra. In each case below the conclusion is not logically entailed, even with common knowledge; it comes from suppositions about the speaker's reasons: why did he say that, why put it that way?
| Exchange | What the hearer infers | Maxim doing the work |
|---|---|---|
| Mary: "Where's Paul?" Peter: "He's in one of the Dakotas." | Peter does not know which Dakota | Quantity: he would have said |
| Gourmet friend leaving: "Don't order the quiche tonight; it's dreadful." | She has tasted it | Quality: she would not damn an untried quiche |
| Son: "I wish I had money for a movie." Father: "The lawn needs to be mowed." | There is money in lawn mowing | Relation: otherwise the remark is off topic |
| Marsha: "John is my mother's sister's husband's son." | John is not Marsha's cousin | Manner: why else avoid the word "cousin"? |
Paul Grice explains these by treating talk as rational interaction: we draw conclusions not only from views expressed but from how people conduct themselves in a joint communicative venture. Since Mary has no reason to doubt Peter's cooperation, she infers he is giving as much information as he can. The extra information arrives through nothing but words: no faces or intonations are needed, so it could all be done by teletype, squarely inside the Turing test.
- Quantity. Provide as much information as will be useful and appropriate.
- Quality. Take care not to say false things.
- Relation. Be relevant: stick to the topic.
- Manner. Be perspicuous: clear, orderly, brief.
The maxims are neither complete nor inviolable, but they give a rough standard that, other things being equal, everyone takes for granted. (The short labels are Grice's own; Haugeland lists the maxims without them.)
Implicatures are called conversational implicatures, and the most striking ones are deliberate transgressions that exploit the maxims. Asked for a reference, I write: "I have known Mr. Frump for six years. He is always quietly pleasant and has no nasty habits." I have flagrantly breached the first maxim, the reader notices, and knows I anticipate his noticing, so he "gets" the hidden message: Frump, though nice, is not particularly bright.
Adding pragmatic to symbolic sense does not clarify "making sense"; it makes the picture messier, with no single unifying theme such as truth, rationality or competence. A coherent conversation is not a random sequence of true utterances or successful speech acts: "truth salad and success salad make as little sense as word salad." What continuity, development and legitimate digression in discourse (and in life) consist in has no precise criteria. Haugeland can do no better than a deliberately homely rule.
The Nonasininity Canon: an enduring system makes sense to the extent that, as understood, it isn't making an ass of itself. The joke carries a serious point: the ultimate standard of intelligence is holistic and open-ended, and no formal criterion yet captures it.
- Definition. Metacognition is cognition about cognitions: beliefs, hopes and fears about other beliefs, hopes and fears, our own or other people's.
- Reading other minds. Understanding implicatures is a special case of inferring others' knowledge and motives from what they do; we also infer from their other actions, from what they tell us, from their situations ("she must wish that bear would leave her alone") and from general knowledge of what "everybody thinks".
- Why it matters. People are the most important, most common and most complicated fixtures in daily life, and anticipating them requires appreciating their points of view, values and goals (the indirect lesson of semantic intrigue). This creates a special version of the frame problem: keeping track of what everyone else now believes.
- Reflection. Many philosophers deem reflection the essence of (self-)consciousness; Haugeland is not persuaded, but reflective self-awareness is central to what we call mature, profound or insightful, and no system incapable of metacognition could reflect.
Pragmatics and theory of mind in language models. Grice's maxims (from his 1967 William James Lectures, published as "Logic and Conversation" in 1975) became the backbone of computational pragmatics, and they reappear almost verbatim in how chat assistants are trained: the goals of being helpful, honest and harmless (Askell and colleagues at Anthropic, 2021) map roughly onto quantity and relation, quality, and the institutional side of speech acts. Large language models handle everyday implicatures like the Dakotas and the reference letter well, because such patterns saturate human text. On metacognition the record is mixed: Michal Kosinski (2023) reported that GPT-class models solve classic false-belief tasks, and Tomer Ullman (2023) showed that trivial alterations to the same tasks (a transparent bag, a container the protagonist can see into) broke them. That split, fluent on the familiar case and jerky on a variant, is the pattern Chapter 5's JERK box predicts.
Mental Images
Three puzzles open the section:
- The cube. A three-inch cube is painted blue on two opposite sides and red on one side bridging them; sliced into twenty-seven one-inch cubes, how many small cubes have exactly one red and one blue side?
- The windows. How many windows are on the back wall of the house you know best?
- The giraffe. Imagine riding a giraffe at full gallop across the savanna, chased: where are your arms?
Most people say they answer by forming and inspecting mental images, not by reasoning in words. For the cube, they picture it sliced and count the small cubes along the two edges where red meets blue (three per edge, so six). The windows question splits respondents into two camps: some imagine standing in the backyard counting visible windows, others walk from room to room counting from inside. The giraffe is neither a puzzle with an answer nor a memory, yet nearly everyone says their arms are around the neck. That cannot be extrapolated from horse or camel riding, since riders do not hug those animals' necks; Haugeland suspects people imagine themselves "into" the situation, face the problem of not falling off, and find a tall neck right in front of them.
Not every question invites imagery. "How many dimes make a dollar?" or "Is arsenic good for hamsters?" do not. Some are intermediate: most adults know a tiger has two ears without imaging and counting, but to say what shape the ears are they will check the image, and how many stripes defeats even the image.
Quasi-linguistic representations are complex symbols whose meanings are jointly fixed by syntactic structure and the interpretations of their simple constituents: English sentences, AI stereotypes, computer programs. GOFAI knows how to mechanise these.
Quasi-pictorial representations "depict" their objects by resembling or mimicking them in some way that is hard to characterise: pictures, scale models and perhaps mental images. Nobody yet knows how to build them into a cognitive theory.
The irony is sharp. Hobbes and Descartes built the symbol-manipulation view by striving to overcome Aristotle's resemblance theory of representation, yet we may need both. The main obstacle is saying what mental images could be. They cannot be literal pictures in the brain: there is no third eye inside the skull, it is dark in there, and surgeons with flashlights have never seen any. But there is no inner eye to read quasi-linguistic symbols either, and none is needed: the computer proves in principle that it suffices to have internal structures preserving the essential features of language (digital syntax, the mode/content distinction, semantic compositionality) plus operations that respect them. Might an analogous reply work for images?
What makes something a picture
A floor plan shows the two essentials:
- Correspondence. Its parts correspond to relevant parts of the house (lines to walls, rectangles to rooms).
- Preserved relational structure. The hall is to the kitchen as the bath is to the bedroom, so the hall rectangle is to the kitchen rectangle as the bath rectangle is to the bedroom rectangle.
The second clause is essential, or anything corresponds to anything: the stairwell to the skin of a pomegranate, the parlour to three seeds scattered inside it, with no structure preserved. Blueprints match houses almost trivially (everything but scale). Maps that show altitude by colour gradation are trickier: the cartographer must keep the relations consistent (if A is shown higher than B and B than C, A must be shown higher than C). How the brain represents imagined distance and shape nobody knows, but probably by neither scale model nor colour code: brains are more like pomegranates than sheets of paper.
Six typical operations (easy on paper, puzzling in a head): - Scanning or panning: moving attention from one part to another. - Rotating: changing orientation. - Zooming: changing scale. - Cut and paste: adding, deleting or repositioning salient pieces. - Recognising and counting salient pieces or features. - Superimposing two images to compare them.
The imaginal field on which images are viewed resembles the visual field: it has a measurable, fuzzy-edged shape and size about the same as vision's, and limited acuity, sharper at the centre but less sharp than sight. Images move, rotate and zoom relative to a fixed field: picture your TV set, then zoom in on the control knobs; they grow bigger and clearer while the rest of the set enlarges out of view. Resolution and field stay constant; visible detail and area vary with magnification.
How can an image rotate without twisting brain tissue, or the mind's eye scan with no socket to roll in? Astonishingly, all these operations seem possible, with interesting limits. Psychologists know this partly from introspection, but mainly from experiments that make time the dependent variable, since it is involved in everything and easy to measure, then vary other factors systematically.
- Design. Roger Shepard and Jacqueline Metzler showed subjects 1,600 pairs of perspective drawings and asked whether each pair showed the same object. Subjects reported rotating one object in their minds until it lined up with the other.
- Result. Reaction time, minus a small constant, was directly proportional to the angle of rotation needed. The natural conclusion: people really rotate images, at a constant angular velocity that varies between people but averages about 60 degrees per second (10 rpm).
- Follow-ups. Images pass through intermediate angles, as if glued to a rigid turntable; larger or more complex images rotate more slowly; images of heavy objects the subject has just handled take time to get up to speed and slow down, as if the image itself had mass. So rotating mental images cannot be much like rotating photographs.
- The best part. Rotation in depth (a wheel seen edge on, with parts passing in and out of view) is far more complicated than rotation in the picture plane (a wheel seen broadside), yet subjects did both at virtually the same rate. They seem to rotate not a copy of the drawing but a full three-dimensional image, perhaps an inner "scale model". One can solemnly note that imagery evolved in a 3-D world; Haugeland still finds it amazing.
- Subjective similarity. Imaging is like perceiving, only fainter and less definite: visualising the back of the house is like looking at it with "the mind's eye", imagining trumpets like hearing them. A natural guess is that imagery uses some of perception's equipment.
- Evidence one: confusability. Imagination (dreams, hallucination) is notoriously mistaken for perception, and experiments show subjects can be fooled the other way too, mistaking perception for imagination.
- Evidence two: interference. Perceiving and imagining in the same modality at once is much harder than in different modalities (seeing while imaging a sound), so they compete for modality-specific resources.
- Evidence three: stroke patients. Certain stroke victims with peculiar visual impairments show exactly analogous impairments in visual imagery, implying the damaged tissue served both.
- The catch. That the experience of seeing a house resembles seeing a picture of it shows at most that the house and the picture resemble each other, not that the experience does. Perception and imagination are in the same boat about quasi-pictorial representation, but that alone does not show which boat they are in.
Despite hundreds, if not thousands of imagery experiments, we know next to nothing about how the brain performs these feats, and there is no artificial analog nearly as compelling as the computer is for quasi-linguistic representation. It remains controversial what, if anything, ultimately distinguishes the two kinds; there is no strong reason to think they are our only modes of representation; we know distressingly little about how they interact (pattern recognition may be a special case of that problem); and the limits of imagination are unclear.
Why imagery matters to GOFAI
Two easy replies fail. "Computers can manipulate images", as computer graphics graphically demonstrates: but not every computer simulation is GOFAI. GOFAI processes are essentially cognitive, more or less rational manipulations of complex quasi-linguistic symbols, and graphics techniques have nothing like that character. "It would not matter anyway, since cognition need not explain everything about the mind": this is the serious question, and the answer depends on whether image manipulation is stupid and low-level or not. If it is, cognitive science can adopt a general tactic.
The Segregation Strategy: assume that cognition and "phenomenon X" (imagery, sensation, feeling) are realised in entirely different mental faculties that interact only through a well-defined cognitive input/output interface. X is then real and important but somebody else's problem, as perception already is. If instead imagination is intimately involved in sophisticated problem solving and daily competence, it becomes "a serious alien intruder in the home territory of GOFAI", and it could be that intelligence is not essentially symbol manipulation.
The beauty of images is that spatial side effects take care of themselves. Imagine yourself astride the giraffe and your feet "automatically" end up near its belly; in a scale model of your living room, put the model couch in the front alcove and it is automatically by the bay window and opposite the door. Nobody computes these results; the shapes do the work.
How far can such mimicry go? Move the model table with a tiny lamp on it and the lamp probably rides along. Leave the model windows unlocked and tiny thieves will not steal the tiny stereo. In imagination we can of course imagine a robbery, but the question is what happens automatically, given the image's structure and the operation performed: the couch-by-the-window effect does, the robbery does not. The interesting case is the lamp.
Haugeland makes two points. First, the lamp-table relation is not purely spatial: the lamp is physically supported. Scale models work because spatial relations are modelled by scaled-down spatial relations and dynamic relations by scaled-down dynamic ones. Brains surely do not map space onto space so simply, so some more elaborate system is at work, and nothing obvious stops it from modelling dynamic relations as well. Second, then: automatic modelling of dynamic side effects may be just what is needed to solve the frame problem. Goldy's Baby box (Chapter 5) would ride along with Papa in an image the way the lamp rides on the model table.
The imagery debate and world models. The controversy Haugeland mentions was the "imagery debate" between Stephen Kosslyn, who defended depictive, picture-like representations (Image and Mind, 1980), and Zenon Pylyshyn, who argued since 1973 that images are propositional underneath and that timing results reflect subjects' tacit knowledge of how things would look. Brain imaging later supported a depictive component: imagery activates early visual cortex, sometimes topographically (Kosslyn and colleagues, Nature, 1995), though the debate never fully closed. The "two camps" in the windows question gained a striking extension when Adam Zeman and colleagues (2015) named aphantasia, the lifelong absence of voluntary visual imagery in otherwise normal people, who solve such tasks by other means. Haugeland's Box 4 anticipated a major research line: Kenneth Craik's 1943 idea of the mind as a "small-scale model" of reality returned as the "intuitive physics engine" of Battaglia, Hamrick and Tenenbaum (2013), which explained human judgements about towers of blocks by probabilistic physical simulation, and as the learned world models and video-generation systems of the 2020s that roll scenes forward in time. Whether such models solve the frame problem or only shift it is open: video models still make objects vanish, merge or pass through one another.
Feelings
By common prejudice, robots have no feelings: rational, perhaps, but coldly so. Fiction is unsure whether that is a blessing (no disruptions of mood) or a curse (no life worth living), but either way it assumes thought and feeling are separate, and AI, insofar as it addresses only cognition, assumes the same. The real questions are could AI accommodate feelings, and does it need to?
The separation goes back to Plato and is well articulated in Hobbes and Descartes. Yet the psychology of affect remains underdeveloped: variables are hard to measure and control, and theories are mostly borrowed from elsewhere, as each discipline recasts feeling in its own image.
| School | Its analysis of a feeling |
|---|---|
| Philosopher | Conceptual analysis: regret-P = know-P plus wish-not-P |
| Behaviourist | S-R construct: fear = agitation plus disposition to flee |
| Psychoanalyst | Clue to the unconscious: revulsion = repressed Oedipal aggression |
| Psychobiologist | Autonomic arousal: lust = fired-up midbrain plus sweaty palms |
| Cognitive psychologist | Symbol structure: jealousy = "own" stereotype with inflated "threat" and "defend" values |
Haugeland's aim is therefore modest: a rough sorting of feelings into seven intuitive categories, to keep the variety conspicuous (lest simple answers appeal) and to frame some tentative remarks. The classification is neither complete nor sharp.
| Category | Examples (from his Table 1) | Distinguishing marks | Varies in | Relation to reason |
|---|---|---|---|---|
| Sensations | seeing green, taste of salt, hunger, tickling, pain, nausea, dizziness, orgasm | Raw inputs with definite location and modality; tied to stimulus and physiology; easily induced in the lab; independent of cognition and of each other (one can be colour-blind but otherwise normal) | Intensity of the stimulus | None; animals could have sensations we lack (sonar) |
| Reactions (passions) | delight, lust, revulsion, fright, horror, rage, amazement, exasperation | Automatic, immediate, involuntary, often visceral, short-lived; no fixed organ (can follow news or a sudden realisation) | Fervor about an event | Recognition needed (you can't be horrified without noticing the bodies); never justified, only appropriate or out of proportion |
| Emotions | gratitude, regret, anger, grief, fear, jealousy, pity, indignation, worry | More measured and discriminating; tied to current beliefs more than current input; build slowly, last longer | Fervor | Can be justified or not, yet hard to control (fear is not fright watered down) |
| Interpersonal feelings | love, friendship, hatred, affection, enmity, devotion, trust | Focused on particular, familiar individuals, appreciated as complete persons; can last a lifetime | Depth: how fully the other is integrated into oneself | Only indirectly responsive; one can love a rascal |
| Sense of merit (others) | respect, contempt, esteem, spite, admiration, scorn, reverence | More specific and detached (I can admire a man's eloquence and scorn his views) | Conviction | Highly relevant, subtle; yet felt, not ratings "of socks or toasters" |
| Sense of merit (oneself) | pride, self-respect, embarrassment, shame, guilt, conceit, modesty, hurt feelings | Like the above but involving the self, which makes all the difference | Conviction | As above; the bridge to ego |
| Moods | elation, cheer, serenity, optimism; melancholy, boredom, grumpiness, the jitters, pessimism | About nothing in particular; colour the whole world at once; "descend on" or "come over" us; last hours to weeks | Some kind of depth | Never rational or justified; infect what we notice, desire, expect and pursue. "Existentialists love them." |
Feelings and AI, category by category
- Sensation. Machines plainly "sense" if that means discrimination: electric eyes, digital thermometers and touch sensors give different symbolic responses to different conditions. But it is hard to imagine they feel anything; the intuition is clearest for pain: systems detect damage and take corrective steps, "but do they ever hurt?" The more Haugeland thinks about what is missing, the less sure he is what the question means (which is not to say it is meaningless). Perhaps it does not matter: if felt quality makes no difference except as a cause of inputs, segregation works, and sensations reach cognition as quasi-linguistic messages like "red here now" or "fierce, stabbing pain in left shin this instant."
- Passions. Robotic rage, delight, lust or revulsion are harder to swallow, yet passions seem more intimately cognitive. A subtler segregation might treat them as compounds of three parts: (1) mammalian physiology (the arousal), (2) the sensation faculty (the affective quality), (3) a genuinely cognitive component (the recognition), interacting bidirectionally through I/O channels. That accounts for arousal and recognition, and the physiological part explains the absurdity of computer lust. Often the cognitive component is supposed to differentiate the passions, but segregation does not need that.
- Emotions and sense of merit. The same "divide and conquer" might extend here, with varying proportions of the components explaining the categories: arousal is less conspicuous in reverence and scorn than in fright and horror, thoughtful estimation the reverse, and gratitude and grief fall in between. Intellect could then do without them all, apart from their cognitive components; compassion and fear would change AI systems, but not necessarily their intelligence.
- Interpersonal feelings. Why is a machine "in love" so alien? Not hormones (blushing is incidental), nor lack of cognition: friendship and devotion look like complex combinations of values, assessments, hopes, plans and commitments. What is distinctive is that they flesh out and give meaning to our own lives; without love or at least enmity, existence would be "a contentless, pointless cognitive skeleton." So interpersonal feelings presuppose personality: to love or hate you have to be somebody.
- Moods. The toughest of all. Not localised to any stimulus, they cannot be inputs ("melancholy here now"); not localised in topic, they have no determinate cognitive component; yet they profoundly affect what and how we think. There is no obvious joint at which to carve, so moods may be an important problem for cognitive science, with unclear implications for GOFAI.
He neither endorses nor rejects composite affects or thought/feeling segregation, and puts little weight on the difficulties raised. He commits to three things only: 1. Affective phenomena are rich and varied. 2. Their relation to intellect and cognition is far from clear. 3. Therefore they deserve a lot more study, even in AI.
Emotion research since 1985. Two developments bear directly on the segregation question. Antonio Damasio's Descartes' Error (1994) described patients with ventromedial prefrontal damage whose intellect tested normal but whose real-life decision making collapsed; his somatic marker hypothesis holds that bodily feeling signals guide reasoning, which is evidence against a clean thought/feeling split (though the hypothesis is still debated). Rosalind Picard's Affective Computing (MIT, 1997) launched a field that detects and models emotion in machines, almost entirely on the "sensation as input" model Haugeland sketches: emotion becomes a classified signal, with no claim of felt experience. Lisa Feldman Barrett's constructionist theory (How Emotions Are Made, 2017) argues emotions are built from core affect plus conceptual categorisation, close to Haugeland's compound picture of passions. LLMs produce fluent emotional language and can be steered into personas, but nothing in their design corresponds to arousal, and whether any machine could hurt is exactly as obscure as Haugeland found it.
Ego Involvement
Catlap asks why the computer moved its queen. Dogleg: to attack my rook and pawn, the key to my defence, to clear a path to my king, to win. "But why is the computer trying to win?" Dogleg mumbles one of four answers: it hopes to impress its new friends; conscience (it donates prize money to Hackers Anonymous); it is a feisty little bastard with an inferiority complex; or it can't help it, it's just built that way. The first replies in the chain seem fine, the last exchange bizarre. Our intuitions conflict: the computer is "trying" to win, yet does not give a damn. There is a difference between having a high goal-value and something really mattering. Could a GOFAI system ever truly care about anything?
Most reasons are instrumental: I go to the store to buy bread to make a sandwich. Such chains cannot go on forever, so some goals must be ultimate, pursued for no further reason. Four sources look easy to automate: biological wiring (hunger, whether I like it or not), whim ("let's take the scenic route"), blind habit and social custom (how you hold a fork, how you pronounce "either"). Are there other ultimate reasons?
Swap the computer for a lanky lad from Brooklyn (Bobby Fischer comes to mind, though he is not named) and ask why he plays to win. Typical answers:
- Recognition. To earn public esteem and so validate his self-esteem (defeat means losing face).
- Pride. The personal satisfaction of accomplishing something difficult, as against frustration with himself.
- Prowess. To show off and prove himself, the underside of vanity being humiliation and fear of inadequacy.
What these share is that his self-image is at stake: his performance reflects on who he is, which matters to him for its own sake, not instrumentally. Ego is a potent motivator, yet "machine and beast alike" lack it, which may be why the computer cannot care. But does intellect need ego? Sometimes evidently not: chess computers lack personal involvement and play excellent chess, and the same holds from pocket calculators to expert systems, each competent and quite selfless. Can AI rest assured that intellect and self can be segregated? Haugeland suspects not.
Three stories that need a self to understand
A friend kept a tame white rat that followed at her heels around campus. Startled by a dog, it fled far up her pantleg and lodged too tightly to back out; she dared not move for fear of crushing it, so she sheepishly let down her jeans, eased out a quivering rodent, and won applause from delighted passersby. We find this funny because we identify with her, share her embarrassment vicariously, and feel relieved it was not us. Embarrassment and relief are possible only for a being with a sense of itself that matters to it and can be awkwardly compromised. The point is not only that her ego was involved then, but that our egos are engaged as we hear the story.
From this Haugeland draws his radical suggestion: that ego involvement may be integral to the process of understanding: actual, current feelings such as embarrassed twinges and vicarious excitement may be essential factors in real-time understanding, not merely background experience. This is not empiricism ("you cannot know what 'embarrassing' means until you have been embarrassed", as you cannot imagine licorice without tasting it). The claim is that listeners' reactions shape what they notice, how they conceptualise and what they are reminded of, and storytellers rely on these responses, as on common sense, for efficiency, disambiguation and effect.
The word "sheepishly" shows how. Common sense rules out "acting like a sheep", but "sheepish" can also mean stupid or meek as well as bashfully embarrassed, and the alternatives get different translations, for instance into German. Ruling out the misreadings rationally would take delicate intellection; we have no trouble because we put ourselves in her place and feel the embarrassment firsthand. A translating machine that got it wrong would not have understood the story.
Ralph asks his new friend Lucifer why someone so brilliant and beautiful was kicked out of Heaven. Lucifer proposes a game: he will sit on a rock while Ralph carries on praising him. Ralph pours out glory and adulation for hours until, bored, he suggests "taking turns". "Ahh," sighs Lucifer, "that's all I said, that's all I said." We first feel with and for Ralph in the irksome adoration game; then the trap springs on us, and we land in Lucifer's shoes, tired of Paradise, sharing his innocent indignation that "our" modest proposal was blown into the crime of the ages by God's infinite vanity. That reaction is taboo; driven to it, we relive Lucifer's impossible rationalisation and the unbearable shame behind it, and so we laugh. The joke works only through the listener's manipulated involvement.
A farmer's son steps on a snake and is fatally bitten. The enraged father chops off the snake's tail with an axe; the snake retaliates by biting all the farm's animals. The farmer brings sweetmeats: "I understand your anger, you understand mine; now that we are even, let's forgive and forget." The snake refuses: "You can never forget your dead son, nor I my missing tail."
The flat version. Aesop could have stated the moral directly: a child is like a part of oneself, such as a limb, because (1) losing one is very bad, (2) once lost it can never be got back, (3) there are no adequate substitutes, and thus (4) they are literally priceless; so treating losses as a "fair exchange" or "getting even" is foolish. That list does not misstate the moral; it lacks it altogether, being "utterly flat and lifeless".
Why the fable lives. We identify with the farmer, are brought up short by the serpent's rebuke, and look at ourselves. The terror of losing one's legs is the thought of a crippled life; is losing a son the same? No: many a parent would joyously give both legs to have a lost child back, since children can mean more to who one is than one's own body. The folly the fable is really about is not knowing who you are and what is truly valuable in your life. Getting a moral is not deducing a proposition; it requires the reader's own self to be engaged.
The objection. A system need not have an ego, only know about one. Imagine Flo, a cognitive system with no ego of her own but a rich, detailed theory of Curly, ego and all; she understands stories intellectually by consulting the theory to see how Curly would react.
Haugeland's reply, reconstructed: 1. No theory could predict Curly's responses without working through his salient intervening thoughts: a person's input/output function has no simpler description than its generating process. 2. So the theory must be a model, "inevitably named Moe", that predicts by simulating Curly in detail. 3. The simulation must include Curly's cognition along with his ego, since cognitive states shape and guide ego involvement. 4. So Flo, being selfless, must hand Moe the whole story and plagiarise his response: her "cognitive understanding with occasional reference to a theory of selves" is a total scam, and Moe does all the work. 5. Does Moe merely simulate Curly, as computers simulate hurricanes? If Moe is based on a GOFAI-type theory, then by medium independence, "to simulate one is to be one": Moe literally understands, and therefore has a self of his own, not just a theory of one. 6. Conclusion: the theory route does not avoid the ego; it relocates it into the model.
The pure-intellect objection
"Sense of humour, religious appreciation and literary refinement should never be confused with pure intelligence." The ideal of a pure intellect, reason with a character and validity independent of living situations, is as old as pure logic and mathematics. But two things count against it. First, AI itself has already tarnished the dream by showing that knowledge representation is critical to commonsense understanding. Second, that discovery could not have been made with toy puzzles and contrived domains; only fully natural discourse reveals the lurking richness of natural understanding. So any test systematically less demanding than Turing's risks "purifying" the hardest problems right under the rug.
The examples are neither transcripts of unrehearsed dialogue nor highbrow literature: perfectly ordinary people understand them without unusual effort, so a system that could not would not have reached ordinary human understanding. And the tactic generalises: real-time ego involvement defeats the segregation of self and intellect on the same grounds that earlier torpedoed the segregation of intelligence and knowledge (Chapter 5). Everyday conversation is fraught with life, and a skilful raconteur engages our emotions and passions, carrying us from optimism through tension to miserable gloom and back. Haugeland's verdict is one of the book's most quoted lines: "GOFAI will not have succeeded until its systems can be gripped and wrenched by the dramas of daytime television."
If no system that isn't somebody can understand ordinary English, GOFAI is not refuted; its problem just gets larger, since AI systems must include artificial egos. Requirements: - Self-representation and reflection, with a full repertoire of metacognitive abilities (needed if its public standing is ever to matter). - A self-image whose status is of peculiar importance and value, yet vulnerable to the vicissitudes of performance and perception: that is what gives ego its motivating power. - A standard and scheme for measuring personal worth, its own and others'. - Continuity and ongoing ownership. No single event is shameful or foolish in isolation, only as an act in the biography of a responsible, historical individual whose character it reveals and can threaten. It is the same me who then bumbled and now blushes. Only a being that cares who it is as a continuing, selfsame individual can care about guilt, folly, achievement, life or death. - Probably more: Haugeland suspects moral integrity, conscience and the courage of one's convictions (or their possibility) belong to the structure of a self, "and who knows what else?"
"Machine and beast alike"? The claim that animals are utterly devoid of ego was contestable even in 1985. Gordon Gallup's mirror self-recognition test (1970) had shown chimpanzees recognising themselves, and later work extended self-recognition claims to orangutans, dolphins, Asian elephants and magpies, though what the test shows about selfhood is debated. Primatologists also describe status striving, reconciliation and something like embarrassment in great apes. None of this rescues GOFAI, but it suggests ego comes in degrees rather than being a uniquely human add-on. The lanky lad from Brooklyn and the friend with the rat also show a limit of method: the argument rests on introspective intuitions about what understanding requires, which Haugeland himself treats warily in the next section.
Do machines give a damn yet? Haugeland's own earlier paper "Understanding Natural Language" (1979) put this section's theme in a slogan: the trouble with AI is that computers "don't give a damn." Large language models put his test cases to work in an unexpected way. They explain the rat story's humour, translate "sheepishly" correctly into German, and state the farmer-and-snake moral with apparent feeling, having absorbed millions of human reactions to similar stories from text. That looks like Flo consulting a theory, except that the "theory" is a statistical model of how people respond, not a GOFAI simulation of one person. The Moe argument therefore does not apply directly: an LLM is not GOFAI (Chapter 3), so "to simulate one is to be one" does not follow. What LLMs plainly lack is Box 6's continuity and ongoing ownership: by default no persistent biography, no standing that a blunder can damage, and a persona that can be swapped by a prompt. Whether that absence limits understanding, as Haugeland predicts, or only limits caring, is one of the live questions about these systems.
The Very Idea
A mock news item: Dr Phyllis Crock, former plastic surgeon and real-estate tycoon, announces artificial wine synthesised entirely from swamp water and coal tar in the Everglades, of outstanding quality at a competitive price, with federal development grants. Reactions fall into four categories, and so do public reactions to AI.
| Attitude | On Crock's wine | On AI | Takes "real" AI for granted? |
|---|---|---|---|
| Enthusiasm | "Put me in for a dozen cases of Crypto-Cabernet." | Pipe dreams of permanent leisure | Yes |
| Abomination | It threatens jobs and palates, "and it surely causes cancer." | Nightmares of super-smart robots usurping the human race | Yes |
| Debunking | Even if identical to fine Bordeaux, it couldn't be wine: not fermented, not from grapes | AI is impossible a priori; however good, it is imitation, counterfeit, fake | No: says it cannot be done |
| Skepticism | "I'll be amazed if it's much tastier than kerosene": the ingredients lack the needed molecules | AI is an empirical hypothesis whose outcome is doubtful | No: says it probably won't be |
Enthusiasm and abomination raise important issues but lie outside the book. Haugeland engages the two that ask whether real AI is possible.
Against the debunkers
Debunking means purporting to prove a priori, from general principles or conceptual intuition and independently of empirical results, that AI is impossible; debunkers therefore declare Turing's test irrelevant. The argument has a standard form:
- Nothing could be intelligent without X.
- No GOFAI system could ever have X.
- Therefore no GOFAI system could ever be intelligent.
Popular values of X are feelings, creativity, personality, learning, freedom, intuition and morality. Once X is named, the support for one or both premises turns out thin: it is not self-evident that nothing could be intelligent without feelings, or that no GOFAI system could be creative, and being certain "down in your bones" doesn't change that.
The most attractive X is consciousness, conspicuous for its absence from this book and from cognitive science's technical literature. Such deafening silence may conceal a nasty little secret. But no other psychological theory sheds much light on consciousness either, and we have so little grip on the phenomena that both premises are again hard to defend. The issue is elusive, not dismissible: "Could consciousness be a theoretical time bomb, ticking away in the belly of AI?"
- Starting point. Daniel Dennett, rare among cognitivist sympathisers in trying to fit consciousness into an AI-like model, begins from access: we can introspect (know what we think) and speak our minds (say what we think); introspection is like "telling oneself".
- The buffer M. Only a small fraction of mental contents is accessible to telling, so Dennett posits a special memory component, M, holding that fragment, an intermediary between perception, verbal expression and, via the inner referee, problem solving, long-term memory and commonsense inference.
- The decisive move. We do not tell what is in M because an "inner light" illuminates it; our faculties have access simply because of how we are built, and that access is consciousness. Talk of inner lights is the poets' and philosophers' description of the access, offered as if it explained something.
- Haugeland's objections. It is "too pat": consciousness has contents, so make it a container, "Presto, a theory!" It leaves unexplained why perceptions, feelings, reflections and verbalisations share one buffer, why careful, rigorous thinking is always conscious, what M has to do with self-control, diffidence, intent, alertness, insight and presence, and, closest to the core, why M is the only buffer that "seems" any way at all.
Suppose someone designs a GOFAI system that is allegedly conscious, emotional and the rest. At some level it consists of inner players exchanging formal tokens by formal rules, and such systems are medium independent. So rewire Los Angeles to support the required interactions and recruit millions of Angelenos to manipulate tokens by those rules. By definition this sprawling monstrosity, Gabriel, is an actual instance of the system. Could Gabriel, the whole city as one unit and not the people in it, be conscious? One gasps; it seems preposterous, which casts doubt on the original claim. That is how thought experiments work: they magnify and sometimes clarify intuitions. But they can be treacherous. Haugeland is as appalled as anyone at a conscious Los Angeles, yet cannot say why, so perhaps the case shows the limits of his imagination rather than of possibility. (The scenario closely parallels Ned Block's "Chinese nation" argument of 1978, in which the population of China simulates a brain.)
Debunkers want to shoot AI down before it takes off, as if the very idea were incoherent, but the prospects for cheap victories are slender. Cognitive science rests on a profound and distinctive empirical hypothesis: that all intelligence, human or otherwise, is realised in rational, quasi-linguistic symbol manipulation. It might be true, false, or partly true. Skeptics concede that it is empirical and won't be settled by "armchair philosophizing" but doubt the outcome; the perplexities of this chapter, and the same intuitions that inspire debunkers, can support moderate hesitation (one can keep an open mind about Gabriel in principle and stay dubious in fact). The answers will come from the laboratory.
The promise and the record
People who know AI are rightly impressed: it epitomises the most powerful and successful approach to psychology ever known, dispatches metaphysical paradoxes, formulates concrete theories of unprecedented scope and detail, and makes many deep technical discoveries, all with a "wave of the future" air. Yet AI has not kept its promise:
- No arguably intelligent machines exist (discounting "micro-experts").
- No plausible fraction of common sense has ever been programmed.
- No system has the linguistic competence of a three-year-old child, and none is in sight.
What should one have expected? In November 1957, eighteen months after the Logic Theorist was announced and six weeks after Sputnik, Herbert Simon set out what he and Allen Newell expected (published with Newell in Operations Research in 1958). Simon added that "there are now in the world machines that think, that learn and that create", whose range would soon be coextensive with the range of the human mind.
| Simon and Newell's prediction (by 1967) | Status in 1985 (Haugeland) | What actually happened |
|---|---|---|
| 1. A computer will be world chess champion, unless the rules bar it | Best machines "barely master rated" | IBM's Deep Blue beat world champion Garry Kasparov in a six-game match in 1997 (thirty years late); computers have never held the official title because the rules do bar them, and engines now far outplay every human |
| 2. A computer will discover and prove an important new mathematical theorem | "No computer has ever come close" | 1976: the Four Colour Theorem proved by Appel and Haken with essential computer case checking (human-designed). 1996: William McCune's EQP prover settled the Robbins problem, open since the 1930s. 2023: DeepMind's FunSearch found new constructions in combinatorics (the cap set problem). 2024: AlphaProof reached IMO silver-medal standard (Chapter 3); 2025: systems from Google DeepMind and OpenAI reached gold-medal level on IMO problems in natural language. Competition problems are not "important new theorems", so this prediction is partly met and still argued over |
| 3. A computer will write music accepted by critics as having considerable aesthetic value | "Machine composition seems mainly to have faded away" | David Cope's EMI (from the 1980s) produced convincing pastiches of Bach and others; from 2020 neural systems (OpenAI's Jukebox, Google's MusicLM, then Suno and Udio in 2023-2024) generate complete songs from text prompts, prompting record-label lawsuits in 2024. Wide use, but critical acclaim for machine-composed work remains disputed |
| 4. Most psychological theories will take the form of computer programs | Computational concepts "very prominent", perhaps a sociological more than technical success | Cognitive architectures (Soar, ACT-R), connectionist and Bayesian models, and since the 2010s deep networks as models of the visual and language systems of the brain; in 2025 a language model fine-tuned on a large set of psychology experiments ("Centaur", Binz and colleagues) was proposed as a general model of human choice behaviour. Computation is the default language of cognitive theory, though not "most theories" in Simon's sense |
But these are not the failed predictions that falsify a theory. When Kepler mispredicts the position of Mars, his theory suffers, because the prediction follows from it. Simon and Newell's forecasts were not consequences of their approach but enthusiasm on its behalf: "They mistook not the world, but the research." The oblique lesson is that in science it is hard to know what to expect, so it is also hard to tell when a field is failing to live up to expectations: difficult problems take time ("if at first you don't succeed..."), and crippling defects are hard to recognise until a better approach exists to compare them with.
Fulton's Folly, and the skeptics who were right
Fulton's Folly is our cultural heritage: the scoffed-at steamboat inventor persevered and prevailed, and nobody wants to be the blinkered naysayer who ends up eating crow. But skeptics don't always lose:
- Perpetual motion. Nobody built one, and the explanation, energy conservation and the law of entropy, proved important.
- Phlogiston. Nobody managed to weigh it, a "failure" that helped establish oxygen.
- Absolute velocity. Nobody measured the Earth's absolute velocity, a step towards relativity theory.
Discovery overcomes misconception as often as ignorance.
- The behaviourist precedent. Before cognitivism, behaviourism reigned almost unchallenged in American psychology, with established methods, tons of confirmed results, specialty journals, textbooks and a coherent philosophy: every institutional earmark of an advanced science. In retrospect it contributed little to understanding behaviour, let alone the mind.
- Kuhn's paradigm. In Thomas Kuhn's terms (1962/1970), a paradigm is a scientific triumph so impressive and so pregnant with unfulfilled possibilities that a research tradition coalesces around it as a model. Thorndike and Pavlov inspired sophisticated work on conditioning in birds, dogs and rats, and in people insofar as we are similar.
- The imposter. Genuine intelligent behaviour involves processes qualitatively different from conditioned reflexes, so behaviourism extended to all psychology became an imposter: a method adequate in one domain parading as adequate where it has no credentials.
- The worry. Cognitivism is behaviourism's natural child: the same commitment to objective experiment, mechanistic explanation and "scientific" psychology, plus the far more powerful apparatus of symbol manipulation, knowledge representation and heuristic control, and now the same earmarks, "enough fancy equipment, jargon, and data to intimidate almost anyone." In Newell, Shaw and Simon's classic programs, and in expert systems, it has a legitimately inspiring model with substantial progress. The open question is whether that limited but real success generalises to intelligence and psychology as a whole.
Cognitive scientists, like their predecessors, have made undeniable contributions, but notably narrow ones beside psychology's pretheoretic scope. The shadows remain: how we recognise familiar faces, let alone the lives engraved in them or Rembrandt's portrayals; how we understand conversational English, let alone metaphors, jokes, Aristotle or Albee; what common sense is, let alone genius, wit or good taste; what character is, let alone schizophrenia, neurosis or moral integrity. On these, cognitivists are as vague and impressionistic as psychologists have always been. "Be patient, we're only beginning" is fair, and promissory notes are legitimate currency in vigorous sciences, but too much deficit spending fuels inflation. Perhaps automatic symbol manipulation is the key to the mind; or perhaps the programmable computer is as shallow an analogy as the trainable pigeon, the conditional branch as psychologically sterile as the conditioned reflex.
Haugeland's final position: skepticism, not debunking. Taking sides is more fun, and "skepticism is intellectual anemia", but sometimes the results are just not in. He is not convinced that GOFAI is impossible, and far from persuaded that it is inevitable: after thirty years, the hard questions remain open.
GOFAI's prospects, judged forty years on. Haugeland's skepticism about GOFAI specifically has been largely borne out. The expert-system boom of the early 1980s was followed by a bust and the "AI winter" of the late 1980s and early 1990s (Chapter 5); Cyc never delivered common sense; symbolic natural-language understanding never reached a three-year-old's competence. Success came instead from approaches his strict definition excludes: connectionism (1986), deep learning from 2012, AlphaGo's defeat of Lee Sedol in 2016, the transformer (2017) and the large language models that by the 2020s far exceed a three-year-old's fluency and, persona-prompted, pass a three-party Turing test (Chapter 0). So the broader "very idea", that machines might genuinely think, is more alive than in 1985, but not in the form of "rational, quasi-linguistic symbol manipulation". His imposter-paradigm question has simply changed target: whether scaling learned models is the key to mind, or another trainable pigeon with impressive institutional earmarks, is now argued in exactly his terms. And his checklist of omissions (pragmatics, imagery, feeling, ego, consciousness) reads today as a fair map of where debate about LLMs concentrates: pragmatics and imagery have yielded the most, feeling, a persistent self and consciousness the least.
Consciousness, the time bomb. Consciousness stopped being absent from cognitive science soon after 1985. Bernard Baars's global workspace theory (1988) and Stanislas Dehaene's global neuronal workspace developed something close to Dennett's access-buffer idea, and Dennett himself expanded his view in Consciousness Explained (1991). An adversarial collaboration pitting global workspace theory against integrated information theory, with results reported from 2023, confirmed and contradicted predictions of both and settled neither. For AI, a 2023 report by Patrick Butlin, Robert Long and colleagues derived "indicator properties" from such theories and concluded that no then-current AI system was a strong candidate for consciousness, while finding no obvious technical barrier to building systems that satisfy the indicators. Haugeland's "who can say?" remains the honest summary.
Critical lens on the chapter. - Selective omissions. The four phenomena are well chosen, but perception, motor skill and learning get no section here, though they turned out to be where non-GOFAI methods first beat GOFAI. Haugeland's segregation strategy treats perception as safely peripheral, which the line of critique Hubert Dreyfus developed at Berkeley, where Haugeland trained, had already questioned. - Debunkers under-described. He names no debunker in the text. The best-known of the period were Hubert Dreyfus (What Computers Can't Do, 1972) and John Searle (the Chinese Room, 1980); Dreyfus's argument from embodied skill and background is closer to Haugeland's own skepticism than the "X" schema suggests, so the debunker/skeptic line is less sharp than the wine parable implies. - The Gabriel intuition. As Haugeland admits, it shows the limits of imagination as much as of possibility, and the same move (Block's Chinese nation, Searle's room) has been argued over for decades without consensus. - "Barely master rated" in 1985 was accurate for the top programs (Belle had reached US master level in 1983), but the rapid improvement of brute-force search on special hardware was already visible, and within twelve years it beat the world champion.
For production AI in a bank, Haugeland's four omissions translate into concrete failure modes to test for. Pragmatic sense: a relationship-manager copilot must catch implicature ("the client says he's 'reviewing all his banking relationships'") and must never make a promise it cannot keep, since speech acts in finance are institutional acts. Ego and continuity: an assistant has no standing to lose, so accountability must sit with named humans and with persistent records that supply the biography the model lacks. Feelings: customer-facing systems that detect vulnerability or distress use the "sensation as input" model, so treat their outputs as classified signals with error rates, not as empathy. Skepticism over debunking: neither "it can't really understand" nor "it obviously understands" is a control; measured performance on realistic, situation-dependent cases is.
Master Takeaways
Haugeland's book adds up to a precise statement of what it would mean for a machine to think, and a sober audit of how far the symbolic programme had got. Its lasting gift is the distinction between mechanism and meaning described at two levels, together with the insistence that the hard problems of intelligence are common sense, context, relevance and caring, not calculation. Most of the questions he left open in 1985 are the ones now argued over about large language models.
The takeaways
- Thinking-as-computation is a theory, not a metaphor. AI's serious claim is that we are computers ourselves in an abstract sense: intelligence consists in rational manipulation of interpreted symbols, realisable in any suitable medium. That makes AI part of psychology and philosophy of mind, not just engineering.
- A computer is an interpreted automatic formal system. Formal (self-contained, digital token games), automatic (players and referee built in) and interpreted (its moves make reasonable sense about something). This definition, not hardware, is what makes "machine reasoning" intelligible.
- The paradox of mechanical reason dissolves by redescription. The same process is mechanical token shuffling under one description and reasoning about meanings under another; the Formalists' Motto ("take care of the syntax, and the semantics will take care of itself") explains why the two descriptions agree.
- Universality makes architecture an empirical question. Babbage's engine, Turing machines, von Neumann machines, LISP and production systems can all compute the same things, yet differ radically in memory access and control. The mind may have an architecture all its own, to be discovered, not assumed.
- The real intelligence is in formulating problems. Heuristic search tamed the combinatorial explosion (about 10^120 chess games), but experimenters did the formulating themselves; choosing the search space, as with the mutilated checkerboard, is where most of the wit lies.
- Common sense cannot be cut into micro-worlds. SHRDLU's fluency depended on a domain stripped of everything needing wit; trading, relevance and situation connect to all of everyday life. The blocks world approximates a playroom "as a paper plane approximates a duck."
- The frame problem is about relevance in time. Keeping beliefs current after an event means noticing salient side effects without checking everything; cheap tests fail because the world is not compartmentalised, sleeping dogs fail because dependencies vary with the situation.
- Interpretation rests on coherence. To interpret is to render an ordered text so it makes reasonable sense; truth is a large part of sense but not all, and the more complex the interpretive scheme, the more text is needed to make its success non-trivial.
- Original versus derivative meaning is the deepest unsolved problem. Words mean only because users mean things by them; semantically active symbols that interact appropriately with no outside user are candidates for original meaning, but which systems really qualify remains open.
- Simulation is not the thing. A computer simulating a hurricane is not windy; GOFAI claims intelligence is symbol manipulation, not that it can be described symbolically. By this test modern neural AI is not GOFAI, even though it runs on digital computers.
- Digital systems buy perfection despite the world. Positive and reliable write/read techniques give margins within which all performances are equivalent, which is why formal systems can grow enormously complex without getting out of control.
- Heuristics can ride on algorithms. A fallible move chooser is an algorithm for computing an estimate; the same routine is described once as infallible procedure and once as fallible judgement, which is the pattern for every model inside an auditable workflow.
- Knowledge must be organised for access, not just storage. Stereotypes (frames, scripts) win by never considering the irrelevant, by constraints, defaults and inheritance, yet stumble on selection without word cues, knowledge with no topic, and tunnel vision.
- Intelligence is jerky before it is graceful. AI systems cruise along sensibly and then bumble ridiculously; whether smooth good sense is just fine-grained rules or something different in kind is an empirical question that patching cannot settle.
- Pragmatic sense is part of language. Speech acts are actions; Grice's maxims of quantity, quality, relation and manner let hearers infer more than is said, so no system that ignores pragmatics truly speaks the language.
- Ascription needs semantic intrigue. Crediting a system with beliefs and goals is justified only when the ascriptions hang together across many explanations (Supertrap), not merely when they fit one behaviour (the mousetrap).
- Imagery may be an alien intruder. Mental rotation runs at about 60 degrees per second, in depth as fast as in the plane, suggesting quasi-pictorial representations that GOFAI cannot yet model; dynamic images might even let side effects take care of themselves.
- Feelings resist tidy segregation, moods most of all. Sensations might enter cognition as inputs, passions as compounds, but moods are about nothing in particular and colour everything, leaving no joint at which to carve.
- Understanding may require a self. Getting a joke or a fable can depend on feeling embarrassment or shame on someone's behalf; a selfless system consulting a theory of selves must run a full simulation, which, if GOFAI, would have a self of its own.
- Turing's test is the right severity. Any test systematically easier than unrestricted conversation risks "purifying" the hardest problems under the rug; GOFAI will not have succeeded until its systems can be gripped by daytime television.
- Debunking is cheap; skepticism is earned. A priori arguments of the form "nothing intelligent lacks X; no GOFAI system has X" rest on thin premises, consciousness included; the right attitude is to look to the laboratory.
- Failed forecasts misjudge research, not the world. Simon and Newell's 1957 predictions were enthusiasm, not consequences of their theory; the chess forecast came true thirty years late (Deep Blue, 1997), the mathematics forecast is partly met, music remains disputed.
- Beware the imposter paradigm. Behaviourism had every institutional earmark of a thriving science and explained little; a narrow real success (heuristic search, expert systems) can parade as adequate for the whole mind.
- Haugeland's verdict has aged well for GOFAI. Symbolic AI did not deliver common sense or child-level language; success came from learning-based methods his definition excludes, so the "very idea" survives in a form he did not foresee.
Frameworks & mental models
| Name | What it says | Use it when |
|---|---|---|
| Paradox of mechanical reason | If a process respects meanings it cannot be mechanical; if mechanical, it cannot reason | Asking whether any machine "really" reasons |
| Mystery of original meaning | Words borrow meaning from minds; something must mean originally | Judging whether a system understands or merely echoes |
| Suitability of a notation | A notation suits a domain when it can solve problems about it, systematically, with allowed steps | Choosing data models and representations |
| Formal system | Tokens, starting positions, rules of legal moves; self-contained and meaning-free | Specifying any rule-governed process |
| Digital system | Positive, reliable write/read techniques with margins of error | Designing systems of record and data integrity |
| Medium independence / formal equivalence | Same system in any medium; equivalent systems map positions and moves one to one | Arguing about multiple realisability or platform migration |
| Algorithm vs heuristic (redescription) | Infallible recipe vs fallible rule of thumb; one routine can be both | Placing a model inside an auditable pipeline |
| Automation Principle | If legal moves are fixed by algorithms, the system can be automated | Deciding what can be fully mechanised |
| Coherence principle | Interpret an ordered text so it makes reasonable sense | Validating any interpretation of outputs or signals |
| Formalists' Motto | Take care of the syntax and the semantics takes care of itself | Formal verification, rule engines, proof checkers |
| Division of semantic labour (Type A / Type B) | Rules guarantee basic sense, players add refinement; type B systems make sense only as a whole | Deciding between proofs and empirical validation |
| GOFAI's two claims | Intelligence comes from reasonable thinking; thinking is inner automatic symbol manipulation | Classifying an AI approach as symbolic or not |
| Homuncular analysis | Explain intelligence by ever dumber cooperating subsystems down to mindless ones | Decomposing complex systems or agents |
| Semantic activity | Symbols that interact appropriately without a user are candidates for original meaning | Assessing claims that models "understand" |
| Turing's thesis and universality | Any automatic formal system has a Turing equivalent; one universal machine can mimic all | Separating what is computable from what is practical |
| Five architectures | Babbage, Turing, von Neumann, LISP, production systems differ in memory access and control | Choosing workflow, functional or rules-based designs |
| Virtual machines | Libraries of subroutines plus a librarian create a new computer | Layered platforms and abstraction |
| Generate and test; means-ends analysis | Generator defines the search space, tester the goal; reduce differences, set subgoals | Planning, search and agent loops |
| Expert-system conditions | Few known variables, computable dependencies, enough complexity | Deciding where rules beat learning |
| Micro-world critique | Contrived domains evade the interconnectedness of common sense | Evaluating curated demos |
| Stereotypes (frames, scripts) | Encyclopedic defaults with constraints and inheritance; fail on selection, no-topic, tunnel vision | Knowledge graphs, ontologies, context design |
| Frame problem (cheap test vs sleeping dog) | Update side effects without checking everything; both schools fail in open worlds | State management, data lineage, impact analysis |
| Ascription schema and semantic intrigue | Ascribe states to maximise competence, and only if they hang together | Attributing intent or goals to systems |
| Grice's maxims | Quantity, quality, relation, manner yield implicatures | Conversational design and evaluation |
| Nonasininity Canon | A system makes sense to the extent it isn't making an ass of itself | Holistic acceptance testing |
| Segregation strategy | Wall phenomenon X off behind an input/output interface | Modular design of perception, affect, tools |
| Seven kinds of feelings | Sensations, passions, emotions, interpersonal, merit (others, self), moods | Affective computing and customer-sentiment design |
| Artificial ego requirements | Self-model, vulnerable self-image, worth scheme, continuity | Thinking about persistent agents and accountability |
| Debunking vs skepticism | A priori impossibility vs empirical doubt | Weighing claims about AI's limits |
| Imposter paradigm | A real but narrow success posing as a general science of mind | Judging whether a method will generalise |
Strengths. The clearest short account ever written of why symbolic AI was philosophically attractive: exact definitions (formal system, digital, algorithm, interpretation, computer), a genuinely illuminating resolution of the paradox of mechanical reason, and a fair-minded treatment that refuses both hype and a priori dismissal. The diagnoses of micro-worlds, stereotypes and the frame problem remain accurate, and coining GOFAI gave the field a lasting name.
Blind spots. The history runs straight from Hobbes to symbolic AI, leaving out the associationist and neural-network line (Hume to perceptrons) that later won; learning is deliberately set aside; perception and motor skill are treated as peripheral even though they were where non-symbolic methods first beat GOFAI. "Making sense" is admitted to be murky and never defined, and the book's own skepticism leans on introspective intuitions (giraffes, embarrassment, Gabriel).
What is dated. Brain figures (10 billion neurons rather than about 86 billion); chess machines "barely master rated" (Deep Blue beat Kasparov in 1997); "no computer has come close to an important mathematical discovery" (the Robbins problem fell to an automated prover in 1996, and AI systems reached IMO gold-medal level in 2025); expert systems presented as promising before their late-1980s bust.
How the field moved. Connectionism (1986), deep learning (2012), AlphaGo (2016), transformers (2017) and large language models produced fluent, commonsense-rich behaviour without the symbol manipulation GOFAI requires. That vindicates Haugeland's doubts about GOFAI while reopening his deeper questions in new form: original meaning and grounding, the jerky reliability of systems that lack a world model, whether understanding needs a self that cares, and whether scaling is a new imposter paradigm.
Who's who
| Name | Who | Why they matter here |
|---|---|---|
| Nicolaus Copernicus | Astronomer (1473-1543) | Split appearance from reality, undermining resemblance |
| Galileo Galilei | Physicist (1564-1642) | Geometry for motion: notation represents whatever it reasons about; secondary qualities |
| Thomas Hobbes | Philosopher (1588-1679) | "Ratiocination is computation": AI's grandfather |
| René Descartes | Philosopher, mathematician (1596-1650) | Analytic geometry; thought as notation; dualism; the machine test that anticipates Turing |
| David Hume | Philosopher (1711-1776) | Mental mechanics by association, losing meaning and reason |
| Charles Babbage, Ada Lovelace | Engineer; his interpreter | Analytical Engine: programmability and conditional branching |
| Alan Turing | Mathematician | Turing machines, universality, Turing's thesis, the imitation game |
| John von Neumann | Mathematician | Stored-program architecture, subroutines, the machine nearly all computers use |
| John McCarthy | AI pioneer | Named AI; invented LISP (trees, recursion); framed the frame problem with Hayes |
| Allen Newell, Cliff Shaw, Herbert Simon | Carnegie-Rand team | Logic Theorist, GPS, heuristic search, production systems, the 1957 predictions |
| Marvin Minsky | MIT AI pioneer | Frames (1974); simplest universal Turing machine of his day |
| Roger Schank | Yale AI researcher | Scripts for stereotyped events |
| Terry Winograd | Stanford/MIT researcher | SHRDLU, the blocks-world conversationalist; later a critic of symbolic AI |
| Norbert Wiener | Mathematician | Cybernetics and negative feedback: a false start for AI |
| Warren Weaver, Yehoshua Bar-Hillel | MT pioneer; MT critic | Machine translation's rise and its fall on "the box was in the pen" |
| Paul Grice | Philosopher of language | Maxims and conversational implicature |
| Roger Shepard, Jacqueline Metzler | Psychologists | Mental rotation (1971), evidence for quasi-pictorial imagery |
| Daniel Dennett | Philosopher | Levels of description; the M-buffer account of consciousness |
| Hubert Dreyfus | Philosopher, AI critic | Simulation-versus-GOFAI distinction; the embodied-skill critique |
| Thomas Kuhn | Historian of science | Paradigms, behind Haugeland's imposter paradigm worry |
Connect it
- Keep the division of semantic labour in production AI. Where a banking domain can be formalised (calculations, eligibility, regulatory thresholds), let rules guarantee correctness and let models make choices; where it cannot (RM conversations, narrative risk summaries), the system is type B and must be validated as a whole by sampling, review and outcome monitoring.
- Treat demos as possible micro-worlds. A copilot that shines on a curated scenario may be SHRDLU on a stage; acceptance tests should use open, messy, situation-dependent cases, the territory where common sense and relevance are needed.
- The frame problem is your data-lineage problem. When one fact changes (an address, a beneficial owner, a sanctions list), which derived facts ride along? Make dependency links explicit outside the model; neither cheap tests nor "let sleeping dogs lie" is safe by default.
- Architecture choices are control choices. Rules engines behave like production systems (easy to add a rule, hard to audit interactions), orchestrated workflows like von Neumann programs, and agentic LLM systems mix both, inheriting both sets of trade-offs.
- Accountability needs a self the model lacks. LLM assistants have no continuing biography or standing that a blunder can damage, so ownership must sit with named humans and persistent records, and pragmatic commitments (promises, advice) must be checked as institutional acts.