Go to Semiotic Review of Books Home Page
Go to SRB Highlights
Go to SRB Archives


SRB

SRB Archives

This review appeared in Volume 4 (2) of The Semiotic Review of Books.

Computers At Work: The Semiotics of Interface Design

by Bernhard Debatin

A Theory of Computer Semiotics. Semiotic Approaches to Construction and Assessment of Computer Systems. By Peter Bogh Andersen. Cambridge: Cambridge University Press, 1990 (Cambridge Series on Human-Computer Interaction 3), 416 pages. ISBN 0-521-39336-1

Computer semiotics is a rather new field within semiotic analysis that increasingly attracts the interest not only of semioticians, but also of computer scientists. As Figge (1991) points out, computer semiotics deals with two different objects, namely computer-aided semiotics and semiotics of the computer as a sign system. Computer-aided semiotics refers mainly to attempts to analyze and to understand the complex sign system of language, such as representation and processing of natural language by means of artificial intelligence. In this domain, formal semantics and linguistics in the tradition of Fodor's and Katz' theory of semantic markers (1964) and Chomsky's theory of generative grammar (1972) have been very successful. Because of the unsolved problems of formalizing natural language -- such as dealing with everyday communication and metaphors -- research in artificial intelligence currently is oriented more toward pragmatic theories of language (see Dreyfus/Dreyfus 1986; with regard to metaphor see my review in SRB 1/1992: "Metaphors and Computers").

In contrast to this, semiotic analysis of the computer as a sign system refers mainly to the relation between computer systems and their human users. In this sense, computer semiotics analyzes structures and functions of the signs that mediate the communication between human and computer. By analyzing the signs of human-machine, computer semiotics systematically can help to find communicative criteria for system design. Therefore even computer scientists increasingly focus on the analysis of communication; the design of interfaces that are both simple and comfortable must consider not only the internal constraints of hard- and software, but also the constraints of human-computer interaction (eg. Winograd/Flores 1986). However, in contrast to computer-aided semiotics, relatively little research has been devoted to the study of computers as sign systems.

This theoretical gap may now be filled by Peter Bogh Andersen's book A Theory of Computer Semiotics.

Andersen's main claim is that his theory of computer semiotics can provide a systematic understanding of the problems of humancomputer interaction and also develop preliminary guidelines for designing computer systems. He defines computer semiotics as "a branch of semiotics that studies the special nature of computer-based signs and how they function in use" (3). In contrast to prevailing approaches in computer science, Andersen thus views computer systems above all as systems that serve specific communicative functions in certain contexts of use. However, these communicative functions can only be served if the computer-based signs are understandable to the user for she or he has no other access to the computer system than through the signs that appear on the screen of the computer. At first glance, this requirement seems to be obvious and easy to fulfil. But as Andersen points out, the interface "can present the signified in a distorted manner, yes even be a lie, and there are many cases where the user is well advised to be suspicious and direct her attention towards the meaning producing mechanism of the interface" (26). Therefore, in order to design an interface that is consistently organized and appropriate to the communicative needs of the user, system development requires a thorough semiotic analysis of human-computer interaction.

In this respect, computer semiotics can support the current paradigm of computer development that mainly focuses on designing a user oriented interface. According to Winograd/Flores (1986), a well-designed interface should reduce the complexity of the computer system without creating new intransparency; i.e. the user should be able to use the system without being forced to deal with problems that don't belong to the actual work task. As Andersen points out in the introduction of his book, this requirement has to be translated into the general rule that "a computer system should not communicate a perspective on work that is (at) cross purposes with that of its users" (16). This basic principle of communicative consistency also implies that the development of computer systems always has to consider the context of use: being part of a work context, a computer system should fit into this context, its logic and its specific ontology. To exemplify this principle, Andersen analyzes both workplace and computer programs. The workplace he studies are a small car repair shop and a large postal bank account office (the 'Postal Giro'), where he has taped utterances of workers in order to interpret them. His semiotic analysis of computer systems focuses on selected writing and painting programs, as well as some video games, and an administrative text system that had been introduced into the Postal Giro.


A Rehabilitation of Glossematics

In order to give his analysis a semiotic foundation, Andersen suggests three main theoretical reorientations for computer semiotics: Firstly, he wants to replace the common 'machine as human' metaphor with a view of 'machine as medium,' which permits the description of computers as virtual sign systems, whose signs "can only exist as real signs in situations where users interpret them"(23). Secondly, he emphasizes that all sign systems, including the computerized ones, basically are anchored in society. This means that the computer as well as the developer and the user have to be seen as part of a broader social socio-cultural structure. Thirdly, he aims to supplant formal linguistics (such as Chomsky's theory) with the structuralist tradition. This means that the structuralistic "principle of immanence" (language has to be seen as a structure sui geneds, i.e. not be analyzed by external means) is taken to be the basic assumption of computer semiotics, as only immanent semiotics can provide a sign-oriented understanding of computer systems.

Working thus within the structuralist framework, Andersen relies on glossematics as his main semiotic tool to analyze language and computer systems. Glossematics is a formal method of analyzing texts, invented by Hjelmslev (1963) and further developed by Greimas (1970) and Halliday (1976) inter alia. It permits a linguistic analysis of texts by cutting the text into smaller and smaller pieces in order to find invariant elements and the smallest meaningful units. However as Andersen points out, linguists have for the most part abandoned glossematics because they perceive it to be too formal and rather imprecise. Andersen now argues that these two perceived disadvantages constitute assets in the realm of computer Semiotics (see 13ff). One reason is the theory is an urgent need for theoretical definitions and formal statements of presuppositions in computer science, which has developed as a mainly practical discipline. The other reason is that lack of precision can be a benefit in a field that is dominated by very concrete and precise methods, because it allows the development of abstract and general principles of design. This attempt to rehabilitate glossematics certainly is an ambitious project. Therefore it is not only interesting to see the author's analysis of computers, but also to follow the way in which he transforms the glossematic approach.

Dealing thus with two different groups of readers (semioticians vs. developers of computer systems), Andersen of course has to convince each of his constituencies that the assumptions and terminologies of the other may prove interesting for them. This is surely no easy task for both semiotic theory and computer science can appear strange and difficulty to understand for someone who is acquainted with only one of the fields. In addition, the book is sometimes almost too rich in examples and excursus. Therefore, although Andersen's short summaries and his advice to skip certain parts are helpful, significant effort is still required of the reader. In the end, however, this work pays off for the book offers a profound and detailed analysis of its topics.

Following the structure of the book I will focus now on some basic principles of Andersen's glossematic approach and on some crucial aspects of his theory of computer semiotics. The book consists of three main parts: (1) Theory: a theoretical foundation in which the author presents his semiotic approach and shows how to apply it to the analysis of computer systems. (2) Computers: a detailed semiotic analysis of computer systems, particularly focused on the communicative functions of computer-based signs, and on a semiotic view of programming. (3) Language, Work and Design: an attempt to show how the theoretical framework can be extended in order to describe and to design computer systems in a practical context of work.


Transforming and Applying the Glossematic Approach

In the first part of the book, Andersen deals with three main problems: firstly, the problem of how symbolic activity can be related to non-symbolic activities such as work; secondly, how to adapt and extend the glossematic theory for the later analysis of computer-based signs; and finally, how to interpret computers as sign-systems.

Symbolic and non-symbolic activities: Since computer systems mainly are used in work contexts, the semiotic analysis must be extended to non-symbolic acts. According to Andersen, this can be achieved by a "materialistic view" of language in which the analytical focus of traditional structuralism (language in general) is replaced by types of language that are connected to specific types of work (work language). Following his definition, work language can be seen as a certain type of a "register" which "is the language used in a particular type of situation with the purpose of supporting or changing its activities"(54). In this way, Anderson grounds his semiotic approach in the analyses of the situational use of language and of work related communication, and not in a general abstract language. Thus emphasizing that structures of language have to be seen as "a post hoc description of regularities of praxis" rather than as invariant structures that determine thinking and doing(see 121), Anderson implicitly imports the insights of modern pragmatic philosophy of language into the structuralist paradigm and thereby marks an important emigration of the structuralist principle of immanence.

Adaptation of glossematic theory: Andersen now explains and discusses the basic terms of glossematic analysis, such as functions of dependency (interdependency, determination, and constellation); dimensions of text analysis dorm vs. substance, expression vs. content, system vs. process, syntagma vs. paradigm and methods for analyzing relations between text units as well as for finding invariant elements (commutation test, whole part-cutting Andersen here follows to a large extent the glossematic concept of Hjelmslev and Halliday, but instead of reproducing the whole sophisticated Glossematic method, he extracts from it the fundamental principle of analyzing relations between elements of language and extends it to work: By focusing on the determination of presuppositions and interdependencies, not only basic text units, but also basic work units -- like tasks and their structural elements can be segmented and interpreted as sign tokens.

The invariant elements that are to be found with this procedure can be categorized as features of a specific "semiotic schema," insofar as they belong to the same situation type of sign usage (see 123). Here the context of use assumes a crucial role: "Schema and use are interdependent and mutually influence each other." (396). It is also this centrality of the dimension of use that forces Andersen to abandon the principle of arbitrariness of signs: From his assertion of the predominance of situational language and work contexts follows the claim that 'The form itself is not arbitrary, since some of the differences introduced in the content purport are motivated by practical tasks of the situation" (72). Indeed, in acknowledging a pragmatic motivation of signs, Andersen departs from a basic principle of structuralism.

Andersen now conceptualizes the close connection between work context and language in a systematic way by introducing the notion of language game. A language game is defined as a linguistic unit that consists of saral utterances which are related to non-verbal actions (see 114). Viewed somatically, it serves as a context-related background paradigm that produces a lot of corresponding paradigms. This concept enables Andersen to extend his linguistic analysis to non-linguistic phenomena since they are structured by and connected with the utterances of a given language game. Thus the analysis of language games plays a crucial role in Andersen's approach, especially in his later semiotic examination of computer-based signs in concrete work contexts (see 346ff.).

Computers as sign systems: With the help of this theoretical framework, Andersen now examines computer systems as sign systems. First of all, he debunks the common view that computers are partners or participants in communication, since they actually can only be a medium of communication. In this definition of the computer as a medium, the practical importance and usefulness of Andersen's approach becomes visible: "A computer system is described as a calculus of empty expression units, some of which can be part of the sign system that emerges when the system is used and interpreted by human users. What can be designed is .only the substance in which the sign system is manifested, not the sign system itself" (120). Computer systems can neither interpret nor intentionally use signs, but only generate sign candidates for the human semiosis.

Accordingly, the main difference between the human sign system and the computer lies in the fact that computer systems presuppose fixed descriptions, wheras human semiotic schemata and sign usage are, as mentioned above, dialectically related, i.e. interdependent. Since computer based sign systems emerge only in a situation of interpretation, the signs of the inteMace and those of the register should at least be similar: "The interface and the work language should belong to the same semiotic schema and not to two separate ones (...), so that screen work and other tasks are interpreted as part of the same work situation" (157). Only if this unity of interface and work language exists, can the user's interpretations become a computer based register; only then are they consistent. The interface itself accordingly can no longer be seen as a component of the system, but must be seen as a relation between system process and the users' interpretation" (171).

The meaning of interface signs therefore has to be differentiated into formal and real meaning. Formal meaning refers to the process and the objects in the computer; real meaning designates objects in the user's work context. A design strategy that focuses on formal meaning views programs as mathematical descriptions of internal processes in the computer. This so called 'product-oriented programming' is appropriate to topics without an existing professional language; the designer should then look for 'suitable metaphors' as means for the interpretation of the interface. In contrast to this, a strategy that includes the real meaning should be chosen in contexts where knowledge, skills and a work language already exist. This 'process-oriented programming' enables the designer to base the descriptions of the system on metaphors belonging to the actual work language.

As Andersen points out, a consistent computer based register can only be achieved if the designer follows these conditions. In sum, this first part of the book provides a very interesting interpretation and broadening of the glossematic approach that leads to an appealing application to the analysis of computer systems.


Sign-Oriented Programming as Practical Computer Semiotics

The second part of the book mainly provides detailed semiotic examination of structures and processes of computer systems as well as some semiotic guidelines for system design. Here Andersen reinterprets the whole process of programming and designing as an activity of sign creation, rather than as engineering. Andersen calls this viewpoint "sign-oriented programming."

The following discussion will deal with three aspects of this part of the book: the analysis of elementary computer based signs, the examination of complex computer based signs, and finally the problem of the illusion of reality on the level of the interface.

Elementary signs: Andersen's analysis of simple computer-based signs provides a typology of signs that is graphically formalized by so-called Petrinets diagrams that allow one to show relations and functions, like determination, interdependency and constellation. The advantage of Andersen's typology of simple signs Hes in the fact that signs are classified not only with regard to their interactive function, but also with regard to their influence on other signs. The typology consists of six categories: Interactive signs, such as the cursor can change their own properties as well as those of other signs. Actor signs, such as a table of contents, indicate the state of the system or parts of it. Controller signs, such as window borders, only influence other signs by changing them. Object signs, such as characters of a text, are changeable by other signs without being able to influence them. Layout signs, such as the landscape in a video game, function as decoration and background. Ghost signs, such as traps in video games, are invisible signs that influence other signs.

The sign-oriented view of programming pays much attention to the interplay between these types of signs, as with as to the problem of how to find reasonable criteria for applying the different types in designing user-oriented interfaces. According to the principle of "layering" computer systems, somputer based signs here can be described on two semiotic levels (see 242): The top layer containing features of the interrace, is the form level, with invariant elements that serve communicative functions. The bottom layer, containing the program and descriptions of its execution, is the substance level, which is concerned with variants. The choice of adequate signs concerns the form level, whereas the computational representation of these signs is a question of the substance level .

Complex signs: Elementary computer-based signs are organized in either concurrent or sequential chains. These chains can be described as chains of dancing steps, i.e. as units of connected syntagmatic and paradigmatic movements in time and space (see 215ff). The metaphor of dance and theatre is also helpful for the analysis of complex signs, since they are seen as combinations of simple signs, corresponding to linguistic phrases or sentences. Andersen gives the following general principle: "Analyze sequential and concurrent chains separately, and describe the individual 'dances' as particular combinations of these two kinds of elements" (248). Based on glossematics and formally depicted with Petri-nets, the author now develops a method of scrutinizing the structure of composite computer-based signs. Firstly, a computer-based sign process can be analyzed by cutting large units of the process into smaller pieces, according to their functional dependencies. Next, the sign process can be synthesized by "describing how the larger units are constructed from smaller units" (257). Andersen then shows in detail how to analyze and synthesize different sequential syntagms (tasks, actions, and indicators), as well as concurrent syntagms (scenes, subview, view, and focus).

The complex sign of subview is the most interesting of all of these features, as it shows an object from a certain perspective. Since the same object can also be showed in different ways with other subviews, the presented object seems to take on the quality of a "solid" inside the computer system (see 282ff). As a result, the computer itself seems to become a human-like sign interpreting system that can choose different ways of designating objects according to the relation between semiotic schema and situation of use. Andersen, however points out that this illusion only works because it is based on an invisible basic sign with a description of the object and formal rules for its transformation; the user then only sees the different transformations of the object but not the basic description of it.

Illusion of reality: The example of subview leads to a broader problem, namely the question of to what degree illusion built into the interface is helpful or misleading for the user. As Andersen shows, the illusion of direct manipulation that is provided by user-oriented interfaces is not always an advantage: On the one hand, it reduces complexity and enables the user to focus her attention on the task instead of on the computer on the other hand, it produces a "cast iron" illusion "that prevents the user from getting ideas for modifying her tool and changing her working conditions because the technical workings of the system are so to speak sealed up" (306). Therefore, Illusion of reality as a designing device has to be used very carefully and it should possibly be complemented by methods of rendering the system unfamiliar so that the user can look behind its surface. Andersen suggests this method referring to the effect of "alienation" (Verfremdung) Bertolt Brecht's theory of theatre.

To summarize, Andersen's semiotic analysis of computer systems provides a lot of insights into the problems of their design. The sign-oriented view of programming constitutes a systematic method of analyzing computer systems for programmers and designers. Giving many concrete examples and tentative guidelines for the process of design, the author demonstrates practically how computer semiotics can be applied to programming.


A Semiotic Evaluation of Sign-Oriented Programming

In the third and most empirical section of his book, Andersen argues for the general relevance of his approach and tries to develop semiotic tools for evualating computer systems with regard to concrete work contexts. Using examples from the work field of the Postal Giro, he examines the relation between computer system and work language. He does this both on a structural level and by following change over time after the introduction of a complex computer system. The extent of empirical study in this section is impressive; for our purposes, however, we will limit discussion to the main theoretical notions that inform Andersen's empirical research. Analyzing two main aspects of language, namely language as interpretation and language as action, Andersen determines the notions of 'semiotic field' and 'language game' as the basic means of this evaluation.

The first aspect, language as interpretation, deals with the fact that different perspectives on the same object or work context are expressed in different sub-languages. These sub-languages embody different interpretations of reality. As a set of distinctive linguistic features that are specifically connected with a work context, such an interpretation can be conceptualized by the notion of semantic field. Andersen defines a semantic field as a "language-motivated paradigm that is also task-motivated" (331). Therefore, the concept of semantic field can be used as a measure of how closely symbolic and non-symbolic actions match. This permits an examination of whether the computer-based register fits the work context to which it is applied. In addition, the study of different semantic fields and their changes within an organization is very useful for designing computer systems suitable to different groups of users.

The second aspect, language as action, leads to the interpersonal function of language and thus to pragmatic approaches of linguistics, such as the speech-act theory of Searle (1969). Yet, instead of focusing on the single utterance, as the speech act theory does, Andersen stresses the syntagmatic aspect of language that appears in groups of related utterances, which he calls, as mentioned above, a language game. In this context, he distinguishes two dimensions of language games, namely their internal structure and their external function. The internal structures concern the question of how utterances are connected and to which type of game they belong.

Andersen gives this definition: "The utterances of language games can be described by sentence frames consisting of focus and background paradigms, and the particular combination of focused and backgrounded paradigms characterizes a specific game type" (355).

The external functions concern the way that utterances of a language game are connected to specific non-symbolic actions. With this functional typology of language games, Andersen here introduces an interesting variation on the many attempts to develop a typology of speech acts (see 357ff). He differentiates between the following types of language games: directives (utterance is presupposed by an action); representations (utterance presupposes an action); coordinatives (utterance and action are interdependent communicatives (utterance and action are independent); and regulatives (utterance is presupposed by an action schema, i.e. a set of tasks. As Andersen then demonstrates, this functional typology can be used to analyze and to evaluate work contexts as well as interface design (see 360ff and 375ff), since it helps to examine the relation between the computer based register, concrete utterances, and work contexts.

With his concessions of semiotic field and language game, Andersen not only provides semiotic means of evaluation, he also presents a further development of the two concepts, which may open interesting perspectives not only for computer scientists but also for semioticians. His typology of speech acts, in particular, introduces a new approach that could prove useful to speech-act theorists.


Desiderata of Computer Semiotics

As Andersen points out, his book is only a first step toward developing a general theory of computer semiotics. His theory mainly provides basic semiotic instruments for analyzing and evaluating system design and work contexts. It is true that Andersen also develops some preliminary guidelines with his approach of sign oriented programming, but as he notes, a computer semiotics that provides practical and concrete rules of designing is still far in the future. Though Andersen has an impressive background in computer programming, his language is still primarily that of the semiotician. The result is that this book is more accessible to semioticians than to computer designers. The emerging field of computer semiotics will need to grapple with the problem of translating between the terminology of semioticians and that of computer designers. A further problem is that a discussion about the theoretical background of computer semiotics has not even begun; for instance, should computer semiotics be based on a single approach or rather on a combination of various theories? Andersen does not deal with this problem, but despite his stated reliance on glossematics alone, he frequently departs from it by borrowing from pragmatic approaches. Therefore future theories of computer semiotics should not only develop feasible design principles, but also entail a meta-semiotic dimension that takes a critical look at the problem of competing semiotic approaches.


References

Chomsky, Noam (1972), Language and Mind. New York: Harcourt Brace Janovic.

Dreyfus, Hubert and Stuart Dreyfus (1986). Mind over Machine. Glasgow: Basil Blackwell.

Figge, Udo (1991). "Computersemiotik." In: Zeitschrift für Semiotik, Bd. 13, Heft 3-4, pp. 321-330.

Fodor, Jerry A. and Jerrold J.Katz (1964). The Structure of Language: Readings in the Philosophy of Language. Englewood Cliffs: Prentice-Hall.

Greimas, Algirdas Julien (1970). Du Sens. Essais Semiotiques. Paris: Edition du Seuil.

Halliday, Michael Alexander Kirkwood (1976). System and Function in Language: Selected Papers. London: Oxford University Press.

Hjelmslev, Louis (1961). Prolegomena to a Theory of Language. Madison, Wisconsin: University of Wisconsin Press.

Searle, John (1969). Speech Acts: An Essay in the Philosophy of Language. Cambridge: Cambridge University Press.

Winograd, Terry and Fernando Flores (1986). Understanding Computers and Cognition. Norwood, NJ: Ablex.

Bernhard Debatin is a research assistant in the philosophy department at the Technical University of Berlin, where he is a member of an interdisciplinary project on socio-cultural problems of using computer-aided design. His research bears upon problems of language and knowledge-processing, and of human-computer interaction from the viewpoints of the philosophy of language, communication theory and semiotics. He is also the author of several articles on metaphor theory and cognitive sciences, as well as on interpersonal and mass communication. He was co-editor, with Dieter Hirschfeld, of Antinomien der Offentlichkeit (i989; ISBN 3-88619-613-5); and, together with Hans J. Wulff, co-editor of Das Telefon im Spielfilm (1991, ISBN 3-89166-146-0).


Go to Semiotic Review of Books Home Page
Go to SRB Highlights
Go to SRB Archives