Go to Semiotic Review of Books Home Page
Go to SRB Highlights
Go to SRB Archives
SRB

SRB Highlights

This review appeared in Volume 8 (3) of The Semiotic Review of Books.

Representing Representations

by Markus F Peschl

Forms of Representation: an interdisciplinary theme for cognitive science. Donald Peterson (ed). 208pp. Exeter UK: Intellect Books, 1996. ISBN 1-871516-34-X

Whenever a human cognitive system interacts with its environment, it is confronted with the highly complex, multidimensional, and dynamical structures of the world. Furthermore, the cognitive system has to solve a wide variety of tasks on different levels of complexity (e.g., from "simple" physical survival by searching for food to complex reasoning and problem solving in science). In other words, the interaction between a human and its environment can be described as the interaction between two complex dynamical systems -- the goal is to establish some kind of stability inside the cognitive system (e.g., survival, epistemological and cognitive stability through successful predictions or manipulations of the environment, etc.) and between these two systems. How is this goal achieved?

Cognitive science and cognitive psychology (and, in fact, most approaches in epistemology) are based on the assumption that, in order to behave adequately in a given environment, the organism must have some kind of representation of (at least) some parts of this environment. These representations are referred to as "internal representations", as it is postulated that they exist "inside the head". The common sense view of this claim suggests that the world (or parts of it) is represented in the form of symbols, propositions, sentences, mental images, semantic networks, etc. (e.g., Posner 1989; Newell et al. 1976, 1980; Kosslyn 1990, 1994) and an algorithm or some manipulation mechanism operates on these representations. These operations result in behavioral externalizations of the organism which (hopefully) lead to the desired stabilities described above. Some behaviors result in a special form of manipulation of theenvironment: they produce representational artifacts. In other words, "external" or "externalized representations" are generated by making use of tools, such as pencils, pens, brushes, printers, etc. ("human print-out facility", A Karmiloff-Smith, p 149).

It is these forms of (external) representations ("FOR") which this collection of papers mainly discusses. Humans make use of them and are confronted with hundreds of these representational forms every day: alphabets, diagrams, drawings, flow-charts, computer programming languages, computer interfaces, symbols, language, musical notes, etc. This book investigates the wide variety, the epistemological status, and the application of mostly external representations. "What is striking is that the acquisition of an appropriate FOR can be epistemically advantageous, and can facilitate understanding, problem solving, calculation and the growth of knowledge. The notation may allow us better to communicate with others, and may serve as an aid to memory, but our present interest is that it may allow us to formulate inscriptions which we can work with and manipulate so as to expand our knowledge" (p7).

As an example on could think of the advantages of using Arabic numerals for calculations rather than Roman numerals; or, using a diagram to describe a certain phenomenon instead of explaining it in written language. The topic of different forms of representation is not only interesting from a philosophical/epistemological and representational perspective, but it also touches on important issues in cognitive science, semiotics, and philosophy of science (e.g., two or more theories describe the same phenomenon; see also p 180ff). Furthermore, it is highly relevant to many situations in modern natural sciences: every form of visualization or graphical representation (in physics, chaos theory, dynamical systems theory, chemistry, biology, computational neuroscience, geography, computer graphics, etc.) is an example in which different forms of representation are transformed into each other in order to facilitate the understanding or perception of a certain phenomenon. The basic idea which stands behind this book is that different forms of representation can be interpreted as alternative representations of the same domain. Due to the complexity of the world and the differences in the tasks/goals fo the cognitive system (see above). It is clear (i) that a certain representation represents only a part of the whole complexity and (ii) that the chosen form of representation has to be appropriate with regard to the task the user of the representation has to accomplish. Thus, it is no wonder that we can employ different forms of representations even if they refer to the same domain. It is one of the tasks/goals of the cognitive system (see above), it is clear (i) that a certain representation represents only a part of the whole complexity and (ii) that the chosen form of representation has to be appropriate with regard to the task the user of the representation has to accomplish. Thus, it is no wonder that we can employ different forms of representations even if they refer to the same domain. It is one of the goals of this book to investigate this relationship between the representational characteristics of a certain FOR and the task it has to fulfill in particular contexts.

As is pointed out in the contribution of B. Hiley and P. Pylkkänen, there is an interesting and very important philosophical issue hidden in the phrase "alternative representation of the same domain" (181). Namely, it raises the problem of the relationship between representation and reality which leads to the debate between realism, anti-realism, and internal realism. Hiley et al. stress that "the phrase Ôalternative representations of the same domainÕ is by no means philosophically innocent in so far as it evokes in our minds a picture of a well-defined, ready-made, mind-independent and -- above all -- representation-independent domain which is then represented in alternative ways. The internal realist position is plausible in emphasising that we have no representation- or mind-independent access to such a domain"(182). As an implication, Peterson asks in which sense alternative FORs represent the same domain of reality (13f). He reaches the epistemologically interesting conclusion that "a FOR does not simply represent a sector of reality, but facilitates tasks and processes of calculation and manipulation, performed by particular users under particular circumstances"(14). Furthermore, Peterson points out that the epistemology of FORs is not simply a matter of addressing correspondence relations between inscriptions and facts, but concerns the mental actions and processes of cognitive systems extended through their use of FORs.

Most of the contributions in this book focus on "external representations", such as diagrammatic representations (for instance H. Tabachneck-Schijf and H. Simon: "Alternative representations and instructional material"; P. Cheng: "Problem solving and learning with diagrammatic representations in physics"), maps (B. Whitby: "Multiple knowledge representations: maps and aeronautical navigation"), geometrical representations (L. Goldstein: "Representation and geometrical methods of problem solving"), linguistic/symbolic structures (T. Potts: "The projection of semantic onto syntactic structures"). As is clearly stated in PetersonÕs introductory chapter as well as in WhitbyÕs paper on maps in aeronautical navigation, we have to abandon the idea of finding a single method for representing the world, as (i) we do not have a direct access to the environment (Kant) and (ii) knowledge is always system-relative in the sense that it is constructed in order to be applied for the solution of a problem or for generating some kind of behavior, which is specific for the particular organism in a particular internal and external situation. In the case of maps this means that a useful map does not have to contain and show all possible information about a certain area (which is epistemologically impossible anyway), but has to provide the information which is relevant for the purpose of the map (a map for flight radio navigation looks different from a road map for cars). Hence,"a map may systematically omit information irrelevant to the purpose at issue, and may emphasise and duplicate relevant information". (17)

The epistemologically interesting conclusion, which is not explicitly drawn by Whitby, is that (external) representations do not necessarily have to correspond in an almost photographic or isomorphic manner to the environmental structures. Rather, they fulfill their purpose even better, if they are representing the environment in a system- and task-relative manner. From this perspective, the traditional idea of representations as almost isomorphic referential entities has to be abandoned in favor of the concept of looking at representations as system- and task-relative strategies or instructions for generating successful behavior. Such a view does not only have implications for AI and cognitive science, but also for the design of human-computer interfaces and cooperative human-machine cognition (77f): the goal is no longer to find a representation which is as accurate and as general as possible, but to construct a set of many different forms of representations, which are purpose-specific, ad hoc, and can be changed according to the problem to be solved and according to the cognitive abilities of the user (Peschl 1995).

In Tabachneck-SchijfÕs and SimonÕs chapter on "Alternative representations of instructional material" the effectiveness of different forms of visual representations (e.g., graphs, diagrams, tables, etc.) in the educational fields of physics and economics is investigated. They argue in favor of a correspondence between external forms of representation and internal representational formats and modes of operation by using examples from the above mentioned fields. Cheng introduces in his chapter about "Problem solving and learning with diagrammatic representations in physics" the concept of "Law Encoding Diagrams"(LEDs) which can facilitate problem solving and learning in quantitative sciences, engineering, and mathematics. Cheng defines LEDs as follows: they are"diagrammatic representations, they encode relations between the variables or terms in a law using the geometric structure of diagrams; they are correct and consistent encodings of the laws. Furthermore, each instantiation of a diagram represents an instance of the phenomenon or one case of the laws. The main reason for encoding the structure of laws in LEDs (i.e., in a diagrammatic format) is that the relations between the variables and terms are made transparent and apparent so that qualitative and quantitative reasoning is supported in an effective way."

Potts discusses in his chapter on the "Projection of semantic onto syntactic structures" an interesting FOR which graphically represents the semantic (and partially syntactic) structure of a sentence in a rather elegant manner (a kind of directed graph). Sloman develops in his "position paper" (118) entitled "Towards a general theory of representations" a critical view of different forms and levels of (syntactic, semantic, and pragmatic) representation. In conclusion he writes: "I suspect that contrary to the standard approach to analyzing properties of a language, like its semantic and properties, by simply writing down sentences and formulars relating that language to things in the world, a truly general theory would have to define semantics in terms of the architecture of the control system that makes use of the particular information-rich state, the functional roles of such states within the architecture, and their relations to actual and possible environmental states. Philosophy needs to be closely allied with engineering design." (138f) Such a view is fairly congruent with recent approaches in connectionism and the related fields in epistemology.

From a semiotic perspective this book is especially interesting, as it addresses central issues of semiotics: how can different (external) representations, signs, symbols, artifacts, etc. account for a single environmental phenomenon? How can these different forms of representation be transformed into each other and in which way do theyrepresent the world (i.e., what is their "meaning")? How do these symbols or artifacts (in the widest sense) get and/or change their meaning? A. Karmiloff-Smith gives in her contribution, "Internal representations and external notations: a developmental perspective", an interesting insight: how this relationship between internal and external representations (and, thus, meaning) is developing in children. She shows that external representations ("notations", artifacts) are not merely externalizations of internal representations. "A developmental perspective shows that children sometimes have relatively clear internal representations of a problem and yet experience serious difficulties in translating these representations into a notation -- an external format that leaves a trace... Internal representations undergo multiple changes or redescriptions, resulting in the same knowledge being internally encoded in different representational format of varying degrees of explicitness and accessibility." (141).

One thing which is missing not only in Karmiloff-SmithÕs article, but in the whole book is a serious in-depth discussion about the cognitive, neuroscientific, and epistemological foundations concerning the processes of interaction occurring between internal and external (forms of) representations. Most of the articles focus on very specific details or examples/applications in which different forms of representations are presented, compared, and empirically investigated. This is a legitimate approach. However, in the context of discussing different forms of (external) representations it would be very helpful to investigate (on a epistemological and conceptual level), how, on the one hand, artifacts are produced by the dynamics of internal representations, and how, on the other hand, these (per se meaningless) artifacts are used, perceived, processed by the nervous system in order to facilitate cognitive processes, problem solving, etc. What could have been shown in more detail in this book is the epistemological status and use of FORs; namely, the goal ofFORs is not to represent the world as accurately as possible, but to provide adequate and helpful triggers for the perceiving representational/cognitive system in a certain context. In other words, a certain external representation does not carry a specific representation of the world -- per se it is a meaningless pattern in the environment. Rather, it has been produced or "designed" in order to perturbate/trigger a certain representation or cognitive process in the perceiving system. As the perceiving cognitive system has its own structure, dynamics, and internal state, there is no guarantee that a certain external input (i.e., a representation in a FOR) will always cause the desired internal representational state. That is why, one has to choose between different FORs in order to find the particular FOR which maximally restricts the space of possible internal representations.

Nevertheless, this collection of papers appears to be extremely important in the current discussion about different forms of representations; e.g., propositional vs. neural/connectionist/sub-symbolic representation), as it introduces a new dimension into this debate: it shows the importance and influence of (different forms of) external representations on the representational dynamics of cognitive systems. This frees the field of cognitive science from studying single brains to studying populations of brains, their interactions and their production and use of (representational or symbolic) artifacts. Of course, there is still a long way to go to understand these "cultural processes" in terms of cognitive processes; PetersonÕs book is certainly a first step into this direction.

References

Kosslyn, S.M. (1990). "Mental imagery." In D.N. Osherson and H. Lasnik (Eds.), An invitation to cognitive science, Volume 2, pp.73--97. Cambridge, MA: MIT Press.

---. (1994). Image and brain. The resolution of the imagery debate.Cambridge, MA: MIT Press.

Newell, A. (1980). "Physical symbol systems." Cognitive Science 4, 135--183.

Newell, A. and H.A. Simon (1976). "Computer science as empirical inquiry: symbols and search." Communications of the Assoc. for Computing Machinery (ACM) 19(3), 113--126. (reprinted in M.Boden (ed.), The Philosophy of Artificial Intelligence, Oxford University Press, 1990).

Peschl, M.F. (1995). "Methodological considerations on modeling cognition and designing human-computer interfaces an investigation from the perspective of philosophy of science and epistemology." Informatica 19, 537--556.

Posner, M.I. (Ed.) (1989). Foundations of cognitive science. Cambridge, MA: MIT Press.

Dr Markus Peschl teaches in the Department of Philosophy of Science, Unit of Epistemology and Cognitive Science, at the University of Vienna (Austria). He is the author of several books and many articles on cognitive modeling ("Methodological Considerations on Modeling Cognition and Designing Human-Computer Interfaces -- an Investigation from the Perspective of Philosophy of Science and Epistemology" in Informatica , Vol 19 (1995) 537-556), connectionism, evolutionary epistemology and sub-symbolic neural representation systems ("The representational relation between environmental structures and neural systems" in Nonlinear Dynamics, Psychology and Life Sciences , Vol 1, No 2 (1997) 99-121).


Go to Semiotic Review of Books Home Page
Go to SRB Highlights
Go to SRB Archives