NordiCHI2000 Proceedings, Stockholm Oct. 23-25, 2000. Electronically published by NordiCHI and STIMDI, www.stimdi.se.

Begoniavej 20

DK - 2820 Gentofte

Denmark

tel. 39 65 84 62

Descriptions of the human mental activity found in current CHI literature are found to suffer from defects derived from behaviorist psychology. A case of alternative description, building on classical, introspective psychology, is shown to present insight highly relevant to the design of human-computer interfaces. The view of introspective psychology is characterized by the jumping-octopus metaphor of the state of consciousness, the site-of-buildings metaphor of a person’s insights, and the splashes-over-the-waves metaphor of a person’s verbal utterances. Introspective psychology indicates the importance to CHI of habit, of inattentive operation of equipment, of individuality of imagery, and of individuality of perception of description forms.

Psychology, behaviorism, stream of thought, association, concept, habit, perception, imagery

It is argued that a dominating part of the argumentation around CHI is based on shallow and defective notions about human thinking. More specifically, in discussions of problems of CHI, human thinking tends to be described in terms of behaviourist psychology, as an activity within a black box described in terms of mental models or a human information processor. It is suggested that the description of human thinking of introspective psychology, as presented in the classical work of William James (James, 1890), offers insights of decisive value to CHI.

As a concrete basis for the claims concerning the argumentation around CHI of the present article, the 68 articles presented in the CHI 96 Conference Proceedings have been reviewed for items that are relevant to human mental activities. The findings may be summarized as follows:

1) Of the 68 articles, only 20 make use of any kind of description of the human mental activity.

2) The dominating form of description of the human mental activity found in the articles is in terms of something had or held by persons. That something is called mental models in 7 articles, knowledge in 7 articles, cognitive maps and meanings of words each in 1 article.

3) The place or container where the something is held is called a working memory in 1 article, long-term memory in 1 article, but is not mentioned in the remaining articles.

4) The few remaining descriptions of the mental activity given in the articles refer briefly to users’ feelings, one article talking about a ‘feeling of "natural" dialogue’, one about ‘a system … can feel mechanical and unfriendly’.

5) The ’mental models’ and ‘knowledge’ of 2) are presented merely as labels, without any further detail that might make them specific and meaningful.

6) Apart from the few references to feelings, the descriptions of the mental activity in the articles include no mention, neither of the experience had by people, nor of their concepts and images, nor of the dynamics of peoples’ thought, i.e. the way a person’s experience at one moment leads to the experience of the next. In terms of classical psychology, what is absent from the descriptions in the articles are such items as the stream of thought, the objects of thought and their fringes, the habitual associations of the objects, and concepts.

What has been found here in the CHI oriented literature is found similarly in the writings of psychologists. As one notable example take the article (Miller, 1995), describing a project to build a ‘system that hopes to process natural languages as people do’. Miller describes language in terms of words, word senses, and linguistic contexts. By Miller’s description meanings and senses are verbal, and linguistic contexts form a definite set, which may be used as the basis of word definitions in dictionaries.

Miller describes a difficulty of his system: ‘Polysemy [words having more than one sense] is a major barrier for many systems that accept natural language input. … Choosing between alternative senses of a polysemous word is a matter of distinguishing between different sets of linguistic contexts in which the word form can be used to express the word sense. People are quite skillful in making such distinctions … How people make such distinctions is not well understood.’

With these words Miller displays an entirely misguided view of the human mental activity. With his notion of linguistic contexts that form a definite set Miller is blind to the fact that word meanings are personal images and feelings, and that linguistic contexts are personal and that they continue to be formed and developed by every person engaged in linguistic activity, by the millions every day. A person’s understanding of a word depends in no way on ‘distinguishing between different sets of linguistic contexts’. That understanding develops by the mental mechanism of association from the person’s total situation. The context is that total situation. The variants recorded in dictionaries merely describe some limited aspects of the way words have been used in selected literary styles. These dictionary descriptions are entirely inadequate in accounting for the understanding achieved in actual conversations. The word ‘he’, for example, is described in Webster’s Dictionary by four variants. In actual conversation it denotes one out of several billions of male persons.

Thus Miller’s difficulty of polysemy reveals his approach to ‘processing natural languages as people do’ to be invalid.

In order to make the present objections to the descriptions offered by behaviouristic psychology specific, let me consider a concrete situation related to computer-human interaction: the work I am engaged in daily, of composing the text of a book, sitting in my workroom before the screen of my computer, using an ordinary text processor.

First, behaviourist psychology: Described by a behaviorist psychologist sitting next to me, my activity consists mostly of a succession of pauses during which nothing much is seen to happen, interrupted by bursts of activity of my fingers dancing on the keyboard and the corresponding characters appearing on the screen. Once in a while I may be seen to grab the mouse and have the cursor move about and select text on the screen.

What does this tell us that might help to design the interface? Nothing.

Next, mental models: What are these mental models supposedly at work while I am composing the text? Who can tell? Certainly not I myself; I have no idea what these people are talking about with their mental models.

Next, the information processing model, for example the GOMS model (Card, Moran, Newell, 1983). According to this model the core of the activity is the contents, called my ‘knowledge’, of a container called memory, these contents coming in four forms, called physical, acoustical, visual, semantic. Now this might at first sight appear to be appropriate, since what I am doing is to produce a text. And so by this description my activity with the text processor is a copying of text held in acoustical or visual form in my memory, with the aid of my finger movements on the keys.

But this description misses the whole point of my activity, which is that I want to produce a text, but I do not have it anywhere. Claiming that the text I want to see is hidden somewhere in a memory within me is nonsense. Even the text I am engaged in developing, that which I have produced today, I cannot recall word by word.

And so the information processing model also misses telling us how the interface with the computer should be designed so as best to support my activity. This model does not even begin to suggest how to make sense of the succession of pauses and finger activity that the behaviourist psychologist will see, and is equally hopeless in making sense of the actual text that gradually appears on the screen.

A designer of a text editor interface who would start from imagining users in terms of an information processor would never arrive at the interfaces that are in successful common use today. With the information processor description there is just ‘knowledge’ in the form of text. So the task would be just to copy that text from one end to the other.

What is missing, more than anything, in any of these descriptions, is any mention of what it is that drives me forward in my activity. Why do I do anything at all? Why do I do one thing rather than another? These descriptions give no hint at how to relate my being active to the total situation in which I find myself.

The objections to the description of my writing activity set forth above are merely samples of the discussion of human thinking, knowing, perception, and language, presented much more fully in earlier papers (Naur, 1992) and in two books (Naur, 1995 and 1999). Likewise, the following alternative description of the activity is based on the discussions of these books. This is a description according to classical psychology (James, 1890). It builds on a description of my experience, my stream of thought, as it is known to me through introspection. I experience my being situated on my chair, surrounded by the things of my room, seeing the complicated panorama of my workplace, with papers lying around and the screen and the keyboard, hearing faint noises coming and going, feeling slight tickles and pressures in various parts of my body. Every aspect of this mental state has a fringe around it, consisting of thoughts of innumerable issues that relate to it and feelings that it arouses in me.

This whole state at any moment is immensely complicated, entirely beyond complete description, and is yet experienced as a whole, in a single pulse of consciousness. The state changes from one moment to the next, being never the same twice.

In this total situation in which I find myself I am able to turn my attention to my current book-writing-activity. This activity, which in James’s manner of speaking is a concept in my stream of thought, has been the permanent anchor of many of my thoughts and activities over the last many months. My activity at the present moment derives its attendant positive feeling of significance from its meaningful relation to that permanent concept.

My mental activity during a pause of finger activity, such as it may be observed by the behaviourist psychologist, has at the center of my attention the meaning of a particular point in the text I am in the process of composing. My mental state is in incessant change, mostly in transitive parts of the stream of thought, the feelings that are part of it undergoing a cycle. During the first part of a pause of my finger work my states are dominated by the feeling of the aching void corresponding to the place in the text on the screen to which I wish to add text. This feeling of aching void is highly specific, characterized by my understanding of the text I am in the middle of composing, and of the still missing portion of it that I wish to enter. While this feeling of void prevails, my state will make rapid transitions, my attention wandering between inarticulate notions of the kind of thing I wish to say and my reading of bits of the text that I have already written. During these rapid transitional states of my thought, far too rapid and numerous to be described in detail, my attention is turned in succession to a number of notions and verbal phrases that share close similarities to that which may fill my aching void. By this direction of my attention I try to activate such habits of mine that tend to call forth such verbal phrases that are associated to the inarticulate notions that engage my attention.

These transitional mental states of mine suddenly change to a substantive state, in which a word or a verbal phrase in spoken form appears to my mental ear. With this substantive state my feeling of void changes to feelings related to the new mental phrase and its fringe of meanings. These feelings accompany new transitive states in which I introspectively examine the verbal phrase. These feelings may be of several kinds. They may be of uncertainty and doubt whether the phrase fills the place of void adequately, in view of the fringe meanings of the verbal phrase. Such a feeling may give rise a new feeling of void and to new transitional states, possibly leading to a new association, giving an alternative verbal phrase, and a decision to reject the first one obtained. Or the feeling may be one of satisfaction with the way the newfound phrase fills my aching void, and if so the mental state may by association lead to the sequence of rapid transitional mental states in which my fingers press the keys.

In terms of this description it is possible to identify some important issues that are of direct relevance to the design of the computer interface. It is clear that the primary issue of my book-writing activity is the generation of verbal formulations, of text. The critical examination of the verbal formulations, which in my activity follows immediately upon the appearance of each verbal phrase, may be considered a secondary matter. Thus the interface should be designed so as best to support the primary generation of verbal formulations. This happens by the mental function of association from such mental objects that habitually have come together with them in my stream of thought. In order to further the desirable associations I should strive to fill my stream of thought with items that in my past experience have become closely associated with what may fill my aching void now. Thus I should turn my attention to such other items, whether they are inarticulate images or verbal phrases in some form. Thus it is clear why my present text editing interface is helpful to me. It lets me pass my attention rapidly over the verbal formulations out of which the still unknown one should come, as the proper extension of the given context.

The wider context of today’s writing activity may be described in terms of my conceptions and images. The core of the activity is my concept, in James’s sense, of my-current-antiphilosophical-book. This concept originated at a quite definite moment, as my response to a debate that I had been engaged in. As a concept in James’s sense it is something that in my stream of thought has a definite identity, something that I may take up at any time knowing that it is the same as what I thought of earlier. It is not something that has its identity defined in terms of particular words or properties. On the contrary, my work consists in fitting it out with words and properties, as in dressing up a mannequin.

While I do not have any experience of ‘knowledge’ in spoken or visual textual form, or any other form, what I do have is imagery of certain kinds. I can imagine, as a vague visual image, the kind of thing I am trying to produce, a document of a certain size, looking like other such documents that I have known in the past. Another set of images I can form shows the intended contents of the text as a set of inarticulate ideas, vaguely clustered around certain cores that perhaps I can label roughly with a few key words. Also I can form an image of the outline of the actual text in progress that I am about to work out, showing vaguely how it is arranged, with certain headings and partly completed text sections, some of them just a sequence of key words and phrases waiting to be worked out in full text.

These images are the substance out of which my writing activity grows. At any odd moment of the day or night I may let them pass in review, activating my accompanying feelings of void, all the time seeking to achieve, by association, new relevant verbal formulations. Whenever such a phrase comes to me I have to record it in writing, as a note, since otherwise it may again slip from my mind. As soon as possible I then enter these new formulations into the text in progress. Then I sometimes find that they are not really new, that they repeat an idea already recorded in the text, usually in other words.

In my formulation activity I can rely on no extended text in my ‘memory’, such as the advocates of the human information processor would claim there is. Not even the text that I am actively engaged in composing, that to which I come back in daily work sessions, is present to me. If it were lost my reconstruction of it would require a substantial new effort.

Knowing a text by heart is a special situation, having acquired the dispositions to rattle off a particular sequence of associations, like a musician playing a piece by heart. This does not imply any special insight into the matter. Facility in memorizing text is highly personal, some have it, some don’t.

In presenting this description of my own experience I do not wish to insist that other people experience the analogous situation in precisely this way. Even so it seems quite obvious that the people who have designed the interface to the text editor which I am using, like those of many of similar editors, have had in mind a user experiencing his or her situation more or less in the way I have just described. This is clear from the way the process is set up, from the presentation before my eyes in a convenient format of a long stretch of the text I have already composed and the availability of convenient ways of making additions and changes at any point of that text.

Most of today’s thinking about human mental life is guided by the notion of the human information processor. The statement is met with, for example in books on cognitive science, that the mind is an information processor. It seems to be overlooked that such a statement at best is no more than a description. The honest way of putting it is to say that the mind may be described as an information processor and leave the metaphysics of the matter to the Aristotelian philosopher. It then becomes clear that there may be other relevant descriptions of the mind. To stir your imagination I would like to present you with a few other such descriptions.

The mental activity is like a jumping octopus in a pile of rags. This metaphor is meant to indicate the way in which the state of consciousness at any moment has a field of central awareness, that part of the rag pile in which the body of the octopus is located. The arms of the octopus stretch out into others parts of the rag pile, those parts presenting themselves vaguely, as the fringe of the central field. The rags of the pile are the mental objects that may come to the conscious awareness. They are of all colors and shapes. The jumping about of the octopus indicates how the state of consciousness changes from one moment to the next.

A person’s insight is like a site of buildings in incomplete state of construction. This metaphor is meant to indicate the mixture of order and inconsistency characterizing any person’s insights. These insights group themselves in many ways, the groups being mutually dependent by many degrees, some closely, some slightly. As an incomplete building may be employed as shelter, so the insights had by a person in any particular field may be useful even if restricted in scope. And as the unfinished buildings of a site may conform to no plan, so a person may go though life having incoherent insights.

A person’s utterances relate to the person’s insights as the splashes over the waves to the rolling sea below. This metaphor is meant to indicate the ephemeral character of our verbal utterances, their being formed, not as a copy of insight already in verbal form, but as a result of an activity of formulation taking place at the moment of the utterance.

As this metaphor suggests, the widespread notion that a person’s language activity involves ‘meanings’ of words held in ‘memory’ is misguided. That this notion continues to be perpetrated is perhaps the most astounding case of current psychological misconceptions, considering the unreasonableness of its implications. Thus by the notion every one of us has in our ‘memory’ the complete set of possible ‘meanings’ of each word. Take for example one of the words ‘he’ or ‘she’. By the notion we have in ‘memory’ ‘meanings’ corresponding to each of the few billion persons in the world, live or dead, that these words might designate, and every time someone talks to us and uses one of these two words our mental machinery supposedly is faced with the choice between these billions of persons. This does not make sense. As every one of us knows, when someone talks to us about a ‘he’ or a ‘she’, what is meant is that one person, male or female, who is being talked about in the context of the conversation. The whole story of meanings of words held in a ‘memory’ is a myth.

All of linguistic activity, speaking and understanding, may be described entirely in terms of habitual associations, precisely as any other actions and perceptions. But as established by William James, what enters into associations is not simple ideas, but the mental objects that appear to us in our stream of thought, each of which is a complicated whole, with a fringe of relations and feelings.

By way of contrast to these metaphors, that of the human mind as an information processor, is highly misleading. This will be clear as soon as the ability of humans to act as or simulate machines is considered. As just one example, a human being trying to act as a machine performing that sort of action in which machines excel, to wit, indefinite repetition of a short pattern of operations, will soon be found to fail. There is significant insight expressed in the stock phrase: mindless repetition.

The failings of the prevailing notions of psychology make these notions misleading as guides to the design of CHI interfaces. Five areas of harmful influences may be mentioned:

Current psychology tends to adopt the ideas of artificial intelligence, as a description of people. Designing CHI interfaces upon such ideas is tantamount to designing, not for real people, but for artificial people.

Describing people in terms of ‘knowledge’ or ‘mental models’ has the consequence that the dynamics of thought, the ways thoughts develop, tend to be ignored. In particular the all-pervading importance of habit on all human activity is lost from sight. Thus in the context of CHI the stress tends to be on the users’ first contact with the interfaces, not on the way the interfaces work for users who have well developed habits in their use. Also the need for acquiring habits tends to be ignored. A habit must be acquired by training, by doing the proper actions in their correct order and manner many times. ‘Many’ means of the order of hundreds. Habits have to be trained and maintained.

The importance of habit displays itself particularly in inattentive use of equipment. Inattentive use is undoubtedly the dominating mode of using computers by anyone who is not a mere beginner. However, as found by Stroem (Stroem, 1996), the inattentive user has to be described as being aware of a certain activity without paying attention to it. This state makes no sense in terms of a human information processor. The development of interface design principles for inattentive operation must build on describing the mental activity in terms of the stream of thought, its objects and their fringes.

Descriptions of people in terms of ‘knowledge’ or ‘mental models’ leave no place for the importance of people’s imagery in their activity. More particularly, the great differences of forms of imagery experienced by various people is lost from sight. The fact that individuals differ greatly in what forms of imagery they experience is a major result of classical psychology which by its very nature is difficult to grasp and to accept. We all tend to assume that as we ourselves experience our stream of thought, so do all other people also.

However, as first found in classical studies by Galton (James, 1890), the imagery that people experience differs enormously from one person to another. For example, some people, notable many scientific persons, have feeble powers of visual imaging.

Galton distinguished between visual, auditory, or motoric images. In an investigation of many scientists from several different fields (Roe, 1953,) it is found that they experience images of the following kinds:

• Concrete visual, usually 3-dimensional

• Visual diagram, geometrical, etc.

• Visualized symbols

• Verbalized formulae etc.

• Verbal imagery

• Imageless thought

• Kinesthetic experience

Undoubtedly, a person’s imagery enters as a decisive part of that persons activities. In particular, the person’s planning of an activity depends on the images of the coming situations the person will have. This is relevant to the design of interfaces to such computer applications that try to help users in their planning.

The perception of various forms of description differs greatly from one person to another. Such individual differences have been demonstrated empirically, in the perception of certain kinds of formal description used as descriptions of computer programs, see (Naur, 1993).

The individuality of the perception of description forms means that a form of description that one person finds highly effective, to another person may be worthless. This is highly relevant to the design of man-computer interfaces that are intended for use by any one other than the designer.

The defects of the behavioristic descriptions of human thinking that currently tend to be adopted by workers in CHI have been displayed. By basing the work on the descriptions of classical empirical psychology, benefits to the field of CHI may be reaped concerning habits, inattentive use of equipment, individuality of imagery, and individuality of perception of description forms.

The helpful comments on drafts of this paper from Erik Frøkjær and Georg Stroem are gratefully acknowledged.