This is a file in the archives of the Stanford Encyclopedia of Philosophy. |

- 1. Introduction
- 2. Diagrams as Representational Systems
- 3. Consequences of Spatial Properties of Diagrams
- 4. Diagrams and Cognition, Applications
- 5. Summary
- Bibliography
- Other Internet Resources
- Related Entries

Challenging a long-standing prejudice against diagrammatic
representation, those working on multi-modal reasoning have taken
different kinds of approaches which we may categorize into three
distinct groups. One branch of research can be found in philosophy of
mind and cognitive science. Since the limits of linguistic forms are
clear to those who have been working on mental representation and
reasoning, some philosophers and cognitive scientists have embraced
this new direction of multi-modal reasoning with enthusiasm and have
explored human reasoning and mental representation involving
non-linguistic forms
[Cummins 1996,
Chandrasekaran *et al*. 1995].
Another strand of work on diagrammatic reasoning shows that there is
no intrinsic difference between symbolic and diagrammatic systems as
far as their logical status goes. Some logicians have presented case
studies to prove that diagrammatic systems can be sound and complete
in the same sense as a symbolic system. This type of result directly
refuted a widely-held assumption that diagrams are inherently
misleading, and abolished theoretical objections to diagrams being
used in proofs
[Shin 1994,
Hammer 1995].
A third direction in multi-modal reasoning has been taken by
computer scientists, whose interest is much more practical than those
of the other groups. Not so surprisingly, those working in many areas
in computer science - for example, knowledge representation, systems
design, visual programming, GUI design, and so on - found new and
exciting opportunities in this new concept of ‘heterogeneous
system’ and have implemented diagrammatic representations in
their research areas.

We have the following goals for this entry. First of all, we would like to acquaint the reader with some details of specific diagrammatic systems. At the same time, the entry will address theoretical issues involved in the topic, by exploring the nature of diagrammatic representation and reasoning in terms of expressive power and correctness. The case study presented below will not only satisfy our first goal but also provide us with solid material for the more theoretical and general discussion in the third section. As mentioned above, the topic of diagrams has attracted much attention with important results from many different research areas. Hence, our fourth section aims to introduce various approaches to diagrammatic research taken in different areas.

For further discussions, we need to clarify two related but distinct
uses of the word ‘diagram’: diagram as internal mental
representations and diagram as external representation. The
following quotation from Chandrasekaran *et al*.
[1995, p. xvii]
succinctly sums up the distinction between internal versus
external diagrammatic representations:

*External diagrammatic representations*: These are constructed by the agent in a medium in the external world (paper, etc), but are meant as representations by the agent.*Internal diagrams or images*: These comprise the (controversial) internal representations that are posited to have some pictorial properties.

These existing diagrams have inspired those researchers who have recently drawn our attention to multi-modal representation. Logicians who participate in the project have explored the subject in two distinct ways. First, their interest has focused exclusively on externally-drawn representation systems, as opposed to internal mental representations. Second, their aim has been to establish the logical status of a system, rather than to explain its heuristic power, by testing the correctness and the expressive power of selective representation systems. If a system fails to justify its soundness or if its expressive power is too limited, a logician’s interest in that language will fade.

In this section, we examine the historical development of Euler and Venn diagrams as a case study to illustrate the following aspects: First, this process will show us how one mathematician’s simple intuition about diagramming syllogistic reasoning has gradually been developed into a formal representation system. Second, we will observe different emphases given to different stages of extension and modification of a diagrammatic system. Thirdly and relatedly, this historical development illustrates an interesting tension and trade-off between the expressive power and visual clarity of diagrammatic systems. Most importantly, the reader will witness logicians tackle the issue of whether there is any intrinsic reason that sentential systems, but not diagrammatic systems, could provide us with rigorous proofs, and their success in answering this question in the negative.

Hence, the reader will not be surprised by the following conclusion drawn by Barwise and Etchemendy, the first logicians to launch an inquiry into diagrammatic proofs in logic,

there is no principled distinction between inference formalisms that use text and those that use diagrams. One can have rigorous, logically sound (and complete) formal systems based on diagrams. (Barwise & Etchemendy [1995], p. 214.)This conviction was necessary for the birth of their innovative computer program

Leonhard Euler, an 18th century mathematician, adopted closed curves to illustrate syllogistic reasoning [Euler 1768]. The four kinds of categorical sentences are represented by him as shown in Figure 1.

Figure 1: Euler Diagrams

For the two universal statements, the system adopts spatial relations
among circles in an intuitive way: If the circle labelled
‘A’ is *included* in the circle labelled ‘B,’
then the diagram represents the information that all A is B. If there
is *no overlapping* part between two circles, then the diagram
conveys the information that no A is B.

This representation is governed by the following
convention:^{[1]}

Every objectxin the domain is assigned a unique location, sayl(x), in the plane such thatl(x) isinregionRif and only ifxis a member of the set that the regionRrepresents.

The power of this representation lies in the fact that an object being a member of a set is easily conceptualized as the object falling inside the set, just as locations on the page are thought of as falling inside or outside drawn circles. The system’s power also lies in the fact that no additional conventions are needed to establish the meanings of diagrams involving more than one circle: relationships holding among sets are asserted by means of the same relationships holding among the circles representing them. The representations of the two universal statements, “All A are B” and “No A is B,” illustrate this strength of the system.

Moving on to two existential statements, this clarity is not
preserved. Euler justifies the diagram of “Some A is B”
saying that we can infer *visually* that something in A is also
contained in B since part of area A is contained in area
B.^{[2]}
Obviously, Euler himself believed that the same kind of visual
containment relation among areas can be used in this case as well as
in the case of universal statements. However, Euler’s belief is not
correct and this representation raises a damaging ambiguity. In this
diagram, not only is part of circle A contained in area B (as Euler
describes), but the following are true: (i) part of circle B is
contained in area A (ii) part of circle A is not contained in circle B
(iii) part of circle B is not contained in circle A. That is, the
third diagram could be read off as “Some B is A,” “Some
A is not B,” and “Some B is not A” as well as
“Some A is B.” In order to avoid this ambiguity, we need to
set up several more
conventions.^{[3]}

Euler’s own examples nicely illustrate the strengths and weaknesses of his diagrammatic system.

Example 1. All A are B. All C are A. Therefore, all C are B.

Example 2. No A is B. All C are B. Therefore, no C is A.

In both examples, the reader can easily infer the conclusion, and this illustrates visually powerful features of Euler diagrams. However, when existential statements are represented, things become more complicated, as explained above. For instance:

No single diagram can represent the two premises, since the relationship between sets B and C cannot be fully specified in one single diagram. Instead, Euler suggests the following three possible cases:Example 3. No A is B. Some C is A. Therefore, Some C is not B.

Euler claims that the proposition ‘Some C is not B’ can be read off from all these diagrams. However, it is far from being visually clear how the first two cases lead a user to reading off this proposition, since a user might read off “No C is B” from case 1 and “All B is C” from case 2.

Hence, the representation of existential statements not only obscures the visual clarity of Euler Circles but also raises serious interpretational problems for the system. Euler himself seemed to recognize this potential problem and introduced a new syntactic device, ‘*’ (representing non-emptiness) as an attempt to repair this flaw. (Letter 105)

However, a more serious drawback is found when this system fails to represent certain compatible (that is, consistent) pieces of information in a single diagram. For example, Euler’s system prevents us from drawing a single diagram representing the following pairs of statements: (i) “All A are B” and “No A is B” (which are consistent if A is an empty set). (ii) “All A are B” and “All B are A” (which are consistent when A = B). (iii) “Some A is B” and “All A are B”. (Suppose we drew an Euler diagram for the former proposition and try to add a new compatible piece of information, i.e., the latter, to this existing diagram.) This shortcoming is closely related to Venn’s motivation for his own diagrammatic system (see Section 3.1 for other shortcomings of Euler’s system).

Venn’s criticism of Euler Circles is summarized in the following passage:

The weak point in this [Euler diagrams], and in all similar schemes, consists in the fact that they only illustrate in strictness the actual relation of classes to each other, rather than the imperfect knowledge of these relations which we may possess, or may wish to convey by means of the proposition. [Venn 1881, p. 510.]

Because of its strictness, Euler’s system sometimes fails in representing consistent pieces of information in a single diagram, as shown above. In addition to this expressive limitation, Euler’s system also suffers other kinds of expressive limitations with respect to non-empty sets, due to topological restrictions on plane figures (see Section 3.1).

Venn’s new system [1881] was to overcome these expressive limitations so that partial information can be represented. The solution was his idea of ‘primary diagrams’. A primary diagram represents all the possible set-theoretic relations between a number of sets, without making any existential commitments about them. For example, Figure 2 shows the primary diagram about sets A and B.

Figure 2: Venn’s Primary diagrams

According to Venn’s system, this diagram does not convey any specific information about the relation between these two sets. This is the major difference between Euler and Venn diagrams.

For the representation of universal statements, unlike the visually clear spatial containment relations in the case of Euler diagrams, Venn’s solution is ‘to shade them [the appropriate areas] out’ ([Venn 1881], p. 122). By using this syntactic device, we obtain diagrams for universal statements as shown in Figure 3.

Figure 3: Venn’s shading

Venn’s choice of shading might not be absolutely arbitrary in that a shading could be interpreted as a visualization of set emptiness. However, it should be noted that a shading is a new syntactic device which Euler did not use. This revision gave flexibility to the system so that certain compatible pieces of information may be represented in a single diagram. In the following, the diagram on the left combines two pieces of information, “All A are B” and “No A is B,” to visually convey the information “Nothing is A.” The diagram below, representing both “All A are B” and “All B are A,” clearly shows that A is the same as B:

In fact, using primary diagrams also avoids some other expressivity problems (to do with spatial properties of diagram objects) discussed below, in Section 3. Surprisingly, Venn was silent about the representation of existential statements, which was another difficulty of Euler diagrams. We can only imagine that Venn might have introduced another kind of a syntactic object representing existential commitment. This is what Charles Peirce did about twenty years later.

Peirce points out that Venn’s system has no way of representing the following kinds of information: existential statements, disjunctive information, probabilities, and relations. Peirce aimed to extend Venn’s system in expressive power with respect to the first two kinds of propositions, i.e., existential and disjunctive statements. This extension was completed by means of the following three devices. (i) Replace Venn’s shading representing emptiness with a new symbol, ‘o’. (ii) Introduce a symbol ‘x’ for existential import. (iii) For disjunctive information, introduce a linear symbol ‘-’ which connects ‘o’ and ‘x’ symbols.

For example, Figure 4 represents the statement, ‘All A are B or some A is B’, which neither Euler’s nor Venn’s system can represent in a single diagram.

Figure 4: A Peirce diagram

The reason that Peirce replaced Venn’s shading for emptiness with the symbol ‘o’ seems to be obvious: It would not be easy to connect shadings or shadings and x’s in order to represent disjunctive information. In this way, Peirce increased the expressive power of the system, but this change was not without its costs.

For example, the following diagram represents the proposition
‘* Either* all A are B and some A is B, *or* no A is B
and some B is not A’:

Reading off this diagram requires more than reading off visual containment among circles (as in Euler diagrams) or shadings (as in Venn diagrams), but also requires extra conventions for reading combinations of the symbols ‘o,’ ‘x,’ and lines. Peirce’s new conventions increased the expressive power of single diagrams, but the arbitrariness of its conventions and more confusing representations (for example, the above diagram) sacrificed the visual clarity which Euler’s original system enjoys. At this point, Peirce himself confesses that ‘there is a great complexity in the expression that is essential to the meaning’ ([Peirce 1933], 4.365). Thus, when Peirce’s revision was completed, most of Euler’s original ideas about visualization were lost, except that a geometrical object (the circle) is used to represent (possibly empty) sets.

Another important contribution Peirce made to the study of diagrams starts with the following remark:

‘Rule’ is here used in the sense in which we speak of the ‘rules’ of algebra; that is, as a permission under strictly defined conditions. ([Peirce 1933], 4.361.)

Peirce was probably the first person to discuss rules of transformation in a non-sentential representation system. In the same way that the rules of algebra tell us which transformations of symbols are permitted and which are not, so should the rules of diagram manipulation. Some of Pierce’s six rules needed more clarification and turn out to be incomplete - a problem which Peirce himself anticipated. However, more importantly, Peirce did not have any theoretical tool - a clear distinction between syntax and semantics - to convince the reader that each rule is correct or to determine whether more rules are needed. That is, his important intuition (that there could be transformation rules for diagrams) remained to be justified.

In [1994], Shin follows up Peirce’s work in two directions. One is to improve Peirce’s version of Venn diagrams, and the other is to prove the soundness and the completeness of this revised system.

Shin’s work alters Peirce’s modifications of Venn diagrams
to achieve an increase in expressive power without such a severe loss
of visual clarity. This revision is made in two stages: (i) Venn-I:
retains Venn’s shadings (for emptiness), Peirce’s
‘*x*’ (for existential import) and Peirce’s
connecting line between *x*’s (for disjunctive
information). (ii) Venn-II: This system, which is proven to be
logically equivalent to monadic predicate logic, is the same as Venn-I
except that a connecting line between diagrams is newly introduced to
display disjunctive information.

Returning to one of Euler’s examples we will see the contrast among these different versions clearly:

Example 3. No A is B. Some C is A. Therefore, Some C is not B.

Euler admits that no single Euler diagram can be drawn to represent the premises, but that three possible cases must be drawn. Venn’s system is silent about existential statements. Now, Peirce’s and Shin’s systems represent these two premises in a single diagram as follows:

In the case of Shin’s diagram, Venn’s shading convention for emptiness, as opposed to Peirce’s ‘o’, much more naturally leads the reader to the inference “Some C is not B” than in the case of Peirce’s diagram.

However, Venn-I cannot express disjunctive information between universal statements or between universal and existential statements. Retaining Venn-I’s expressive power, Venn-II allows diagrams to be connected by a line. Peirce’s confusing looking diagram above is equivalent to the following Venn-II diagram:

In addition to this revision, Shin [1994] presented each of these two systems as a standard formal representation system equipped with its own syntax and semantics. The syntax tells us which diagrams are acceptable, that is, which are well-formed, and which manipulations are permissible in each system. The semantics defines logical consequences among diagrams. Using these tools, it is proven that the systems are sound and complete, in the same sense that some symbolic logics are.

This approach has posed a fundamental challenge to some of the
assumptions held about representation systems. Since the development
of modern logic, important concepts, e.g., syntax, semantics,
inference, logical consequence, validity, and completeness, have been
applied to sentential representation systems only. However, none of
these turned out to be intrinsic to these traditional symbolic logics
only. For *any* representation system, whether it is sentential
or diagrammatic, we can discuss two levels, a syntactic and a semantic
level. What inference rules tell us is how to manipulate a given
unit, whether symbolic or diagrammatic, to another. The definition of
logical consequence is also free from any specific form of a
representation system. The same argument goes for the soundness and
the completeness proofs. When a system is proven to be sound, we
should be able to adopt it in proofs. In fact, much current research
explores the use of diagrams in automated theorem proving
(see Section 4 and
[Barker-Plummer & Bailin 1997,
Jamnik *et al*. 1999]).

It is interesting and important to notice that the gradual changes made from Euler Circles through to Shin’s systems share one common theme: to increase both the expressive and logical power of the system so that it is sound, complete, and logically equivalent to monadic predicate logic. The main revision from Euler to Venn diagrams, introducing primary diagrams, allows us to represent partial knowledge about relations between sets. The extension from Venn to Peirce diagrams is made so that existential and disjunctive information may be represented more effectively.

Both Venn and Peirce adopted the same kind of solution in order to achieve these improvements: to introduce new syntactic objects, that is, shadings by Venn, and x’s, o’s, and lines by Peirce. However, on the negative side, these revised systems suffer from a loss of visual clarity, as seen above, mainly because of the introduction of more arbitrary conventions. The modifications from Peirce to Shin diagrams concentrate on restoring visual clarity, but without loss of expressive power.

Hammer and Shin take a different path from these revisions: To revive Euler’s homomorphic relation between circles and sets -- containment among circles represents the subset relation among sets, and non-overlapping of regions represents the disjoint relation -- and at the same time, to adopt Venn’s primary diagrams by default. On the other hand, this revised Euler system is not a self-sufficient tool for syllogistic reasoning, since it cannot represent existential statements. For more details of this revised system, refer to [Hammer & Shin 1998].

This case study raises an interesting question for further research on diagrammatic reasoning. Throughout the different developments of Euler diagrams, increasing its expressive power and enhancing its visual clarity seem to be complementary to each other. Depending on purposes, we need to give priority to one over the other. Hammer and Shin’s alternative system provides a simple model for the development of other efficient non-sentential representational systems, a topic that has been receiving increasing attention in computer science and cognitive science.

A particular distinguishing feature of diagrams is that they obey certain “nomic” or “intrinsic” constraints due to their use of plane surfaces as a medium of representation. The idea is that sentential languages are based on acoustic signals which are sequential in nature, and so must have a compensatingly complex syntax in order to express certain relationships - whereas diagrams, being two-dimensional, are able to display some relationships without the intervention of a complex syntax [Stenning & Lemon 2001]. Diagrams exploit this possibility - the use of spatial relations to represent other relations. The question is; how well can spatial relations and objects represent other (possibly more abstract) objects and relations?

Logical reasoning with diagrams is often carried out in virtue of
their depiction of all possible models of a situation, up to
topological equivalence of the diagrams (this, of course, depends on
the particular diagrammatic system in use). A single diagram is often
an abstraction over a class of situations, and once a suitable diagram
has been constructed, inferences can simply be read-off the
representation without any further manipulation. In some diagrammatic
systems (e.g., Euler Circles) inference is carried out by constructing
diagrams correctly and reading information off them. The complexity
of using inference rules in a symbolic logic is, in these cases,
replaced by the problem of drawing particular diagrams
correctly.^{[4]}
For instance, an Euler Circles diagram ventures to capture
relationships between sets using topological relationships between
plane regions in such a way that that it depicts all the possible ways
that a certain collection of set-theoretic statements could be true.
This has two important consequences: (1) if a certain diagram cannot
be drawn then the described situation must be impossible (termed
“self-consistency”), and (2) if a certain relationship between
diagram objects must be drawn, then the corresponding relation can be
inferred as logically vaild. (See the numerous examples in
Section 2.)
This phenomenon is often termed a “free-ride”
[Barwise & Shimojima 1995].
This style of diagrammatic reasoning is thus dependent on a
particular representational use of diagrams - that they represent
classes of models. If a particular class of models cannot be
represented by a diagrammatic system, then those cases will not be
taken into account in inferences using the system, and incorrect
inferences might be drawn. This fact makes the representational
adequacy of diagrammatic systems, restricted by their spatial nature,
of paramount importance, as we shall now explore.

The representational use of the spatial relations in the plane
constrains diagrammatic representation, and therefore reasoning with
diagrams, in certain important ways. In particular, there are
topological and geometrical (let us lump them together as
“spatial") properties of diagrammatic objects and relations
which limit the expressive power of diagrammatic systems. For
instance, in graph theory it is known that some simple structures
cannot be drawn in the plane. For example, the graph
*G*_{5} is the graph consisting of 5 nodes, each
joined to the other by an arc. This graph is non-planar, meaning
that it cannot be drawn without at least two of the arcs crossing.
This is just the sort of constraint on possible diagrams that limits
the expressive power of diagrammatic systems. Now, since
diagrammatic reasoning can occur by enumeration of all possible
models of a situation, this representational inadequacy (a type of
incompleteness) renders many diagrammatic systems incorrect if they
are used for logical reasoning (e.g., see the critque of
[Englebretsen 1992] in [Lemon & Pratt 1998]).

Perhaps the most simple example of this is due to Lemon and
Pratt^{[5]}
(see e.g., [1997]).
Consider Euler Circles -- where convex regions of the plane represent
sets, and overlap of the regions represents non-empty intersection of
the corresponding sets. A result of convex topology known as
Helly’s Theorem states (for the 2 dimensional case) that if every
triple of 4 convex regions has a non-empty intersection then all four
regions must have a non-empty intersection.

To understand the ramifications of this, consider the following problem:

Note that, in terms of set-theory, only trivial consequences follow from these premises. However, an Euler diagram of the premises, such as Figure 5, leads to the incorrect conclusion thatExample 4. Using Euler Circles, represent the following premises:ABC

BCD

CDA

Figure 5: An Euler’s Circles representation exhibiting Helly’s Theorem

In other words, a user of Euler Circles is
forced^{[6]}
to represent a relationship between the sets which is not logically
necessary. This means both that there are logically possible
situations which the system cannot represent, and that a user would
make incorrect inferences if they relied on the system for reasoning.
More generally, this type of result can be generated for many
different types of diagrammatic system, depending on the particular
spatial relations and objects which they use in representation - a
research programme which is ongoing.

For example, using non-convex regions (e.g., “blobs” instead of circles) leads to a similar problem, only that non-planar graphs are involved instead of Helly’s Theorem. A similar result concerns linear diagrams for syllogisms [Englebretsen 1992], where lines are used to represent sets, points represent individuals, point-line intersection represents set-membership, and intersection of lines represents set-intersection. Again, planarity constraints restrict the expressive power of the system and lead to incorrect inferences.

Atsushi Shimojima’s “constraint hypothesis” perhaps best sums all this up:

Representations are objects in the world, and as such they obey certain structural constraints that govern their possible formation. The variance in inferential potential of different modes of representation is largely attributable to different ways in which these structural constraints on representations match with the constraints on targets of representation ([Shimojima 1996a, 1999]).

As discussed above, much of the interest in diagrams has been
generated by the claim that they are somehow more “effective"
than traditional logical representations for certain types of task.
Certainly, for example, a map is a greater aid to navigation than a
verbal description of a landscape. However, while there are certainly
psychological advantages to be gained through the use of diagrams,
they are (as in the case of Euler Circles) often ineffective as
representations of abstract objects and relationships. Once a purely
intuitive notion, non-psychological claims about “efficacy” of
diagrammatic systems can be examined in terms of standard formal
properties of languages
[Lemon *et al*. 1999].
In particular, many diagrammatic systems are self-consistent,
incorrect, and incomplete, and complexity of inference with the
diagrams is NP-hard. By way of contrast, most sentential logics,
while able to express inconsistencies, are complete and correct.

On the other hand, not being able to represent contradictions could provide us with interesting insights about the nature of diagrammatic representation. If a central goal of a language is to represent the world or a state of affairs, then representing contradictions or tautologies is called into question. Neither contradictions nor tautologies are part of the world. How can we draw a picture, or take a picture, of the contradiction that it is raining and it is not raining? How about the picture of it is either raining or not raining? Now, we seem to be much closer to Wittgenstein’s classic picture theory of language [Wittgenstein 1921].

As a more practical side of the project, AI researchers, one of
whose main concerns is the heuristic power of a representation system
in addition to its expressive power, have been debating for decades
about different forms of representation
[Sloman 1971,
1985,
1995].
Hence, they have welcomed discussions of the distinct role of visual
reasoning and have recently hosted interdisciplinary symposiums on
diagrammatic reasoning at AI
conferences.^{[7]}
At the same time, realizing that human beings adopt different
representation forms depending on the kinds of problems they face,
some AI researchers and design theorists have practiced
domain-specific approaches to bringing in problem-tailored
representation
forms.^{[8]}

For instance, Harel
[1988]
invented highgraphs to represent system
specifications in computer science. This idea has been taken up in
industrial applications (e.g., UML
[Booch *et al*. 1998]).
Several authors have also worked on the problem of automating
reasoning with diagrams in mathematical contexts. For instance, that
the sum of the first n odd natural numbers is n squared is easily seen
by decomposing an n x n grid into “ells”
[Jamnik *et al*. 1999].
Getting computers to carry out this kind of analogical reasoning is
also the task of
[Barker-Plummer & Bailin 1997]
amongst others.

It should also be mentioned that scientists such as chemists and physicists also use diagrams in order to perform certain computations. Feynman diagrams, for example, are used to perform calculations in sub-atomic physics. In Knot Theory (which has applications in physics [Kauffman 1991]) the three Reidemeister Moves are diagrammatic operations which make up a complete calculus for proving knots equivalent.

Do our mental representations have diagram-like or picture-like entities as components? This question has a long history both in philosophy and in psychology, independently of each other. More recently, however, some philosophers have participated in this “imagery debate”, one of the most time-honored controversies in psychology, and some cognitive psychologists find certain epistemological theories in philosophy useful to support their views on the issue.

The nature of mental representation has been one of the perennial
topics in philosophy, and we can easily trace back philosophical
discussions on images and mental representation to ancient
times.^{[9]}
The writings of Hobbes, Locke, Berkeley, and Hume concern themselves
in large part with mental discourse, the meaning of words, mental
images, particular ideas, abstract ideas, impressions, and so on.
Descartes’ well-known distinction between imagining and
conceiving something has generated much discussion about the unique
role of visual images in mental representations. The development of
cognitive science in the 20th century naturally has brought certain
group of philosophers and psychologists closer and we find a number of
authors whose works easily belong to both disciplines
[Block 1983,
Dennett 1981,
Fodor 1981].

Imagery based on mental inspection was the main focus in the early development of psychology until the behavioristic approach became predominant in the discipline. During the era of behaviorism, anything related to mental inspection, including images, was excluded from any serious research agenda. Finally when the topic of mental images made a comeback in psychology in 1960s, researchers adopted a more humble agenda for mental imagery than before: Not all mental representations involve imagery, and imagery is one of many ways of manipulating information in the mind. Also, thanks to the influence of behaviorism, it is acknowledged that introspection is not enough to explore imagery, but a claim about mental imagery needs to be confirmable by experiments in order to show that we successfully externalize mental events. That is, if what a certain mental introspection tells us is genuine, then there would be observable external consequences of that mental state.

Thus the contemporary imagery debate among cognitive scientists is
about the claim that picture-like images exist as mental
representations and about how we interpret certain
experiments.^{[10]}

Kosslyn [1980, 1994] and other pictorialists [Shepard & Metzler 1971] present experimental data to support their position that some of our mental images are more like pictures than a linear form of language (for example, natural languages or artificial symbolic languages) in some important aspects, even though not all visual mental images and pictures are of exactly the same kind. By contrast, Pylyshyn [1981] and other descriptionalists [Dennett 1981] raise questions about the picture-like status of mental images and argue that mental images are formed out of structured descriptions. To them, mental images represent in the manner of language rather than pictures and, hence, there are no picture-like visual mental images.

Both sides of the debate sometimes used a philosophical theory as a supporting factor. For example, pictorialists in the imagery debate found the modern sense-datum theory in philosophy quite close to their point of view. By the same token, the critics of the sense-datum theory argued that the mistaken pictorial view of mental images arises mainly from our confusion about ordinary language and claimed that mental images are epiphenomena.

Without being heavily involved in the imagery debate, some researchers have focused on a distinct role that diagrams or pictures - as opposed to traditional sentential forms - play in our cognitive activities. Based on the conjectures that humans adopt diagrammatic or spatial internal mental representations in their reasoning about concrete or abstract situations (see [Howell 1976, Sober 1976]), some cognitive scientists have concentrated on the functions of images or diagrams in our various cognitive activities, for example, memory, imagination, perception, navigation, inference, problem-solving, and so on. Here, the distinct nature of “visual information,” which is obtained either through internal mental images or through externally drawn diagrams, has become a major topic of research. Even though most of these works assume that there are mental images (that is, they accept the pictorialists’ claim), strictly speaking they do not have to commit themselves to the view that these images exist as basic units in our cognition. Descriptionalists do not have to discard discussions of the functions of images, but only need to add that these images are not primitive units stored in our memory, but formed out of structured descriptions more like the sentences of a language. (See [Pylyshyn 1981].)

A search for the distinct role of diagrams has led researchers to explore the differences among different forms of external or internal representations, and mainly between diagrammatic and sentential representations. Many important results have been produced in cognitive science. Starting from Larkin and Simon’s classic case study [1987] to illustrate a difference between informational and computational equivalence among representation systems, Lindsay’s work locates where this computational difference lies, which he calls a ‘non-deductive’ method. As briefly pointed out above, this inference process is called ‘free ride’ by Barwise and Shimojiima [1995], i.e., the kind of an inference in which the conclusion seems to be read off automatically from the representation of premises. In Gurr, Lee, and Stenning [1998] and Stenning and Lemon [2001], there is an explanation of the uniqueness of diagrammatic inference in terms of a degree of ‘directness’ of interpretation, and it is argued that this property is relative, and hence that “some rides are cheaper than others”. Having the role of graphs in mind, Wang and Lee ([1993]) present a formal framework as a guideline for correct visual languages. At this point, we are very close to applied aspects of research in multi-modal reasoning - design theory and AI research - by providing these disciplines with with computational support for visual reasoning.

Related to the issue of imagistic mental representation is the examination of the semantics of various diagrammatic systems and what they can teach us about the nature of languages in general (e.g., Goodman [1968]). For instance, Robert Cummins [1996], amongst others, argues that too much attention has been given to sentential representations and that focus on a notion of “structural representation” more akin to diagrammatic representation can help to explicate the nature of representation itself. We believe that the considerations presented above give us some empirical handle on this type of claim at least - depending on the imagistic objects and relations used, patterns of incorrect inference should be predictable and detectable. An important article, if little-known, article on this theme is [Malinas 1991]. Here Malinas explores the concepts of pictorial representation and “truth in” a picture via the notion of resemblance, and considers various semantic puzzles about pictorial representation. He develops Peacocke’s “Central Thesis” of depiction ([Peacocke 1987]), where experienced similarities between properties of pictorial objects and their referents in the visual field give rise to the relation of depiction. He goes on to provide a formal semantics for pictures which is “analogous to a semantics for an ideal language”.

- Allwein, G., and Barwise, J., (eds.), 1996,
*Logical Reasoning with Diagrams*. Oxford: Oxford University Press. - Barker-Plummer, D., and Bailin, S., 1997, "The
Role of Diagrams in Mathematical Proofs",
*Machine GRAPHICS and VISION*, 6(1): 25-56. (Special Issue on Diagrammatic Representation and Reasoning). - Barwise, J., 1993, "Heterogeneous reasoning",
in G. Mineau, B. Moulin, and J. Sowa, (eds),
*ICCS 1993: Conceptual Graphs for Knowledge Representation*, volume 699 of*Lecture Notes in Artificial Intelligence*, pages 64-74, Berlin: Springer Verlag. - Barwise, J., and Etchemendy, J., 1989,
"Information, Infons, and Inference", in Cooper, Mukai, and Perry,
(eds),
*Situation Theory and its Applications*, volume 1, Stanford: CSLI Publications. - -----, 1991, "Visual Information and Valid
Reasoning", in Zimmerman and Cunningham, (eds),
*Vizualization in Teaching and Learning Mathematics*, pages 9-24. Washington: Mathematical Association of America. - -----, 1993,
*The Language of First-Order Logic*. Stanford: CSLI Publications. - -----, 1994,
*Hyperproof*. Stanford: CSLI Publications. - -----, 1995, "Heterogeneous Logic", in
J. Glasgow, N. Hari Narayanan, and B. Chandrasekaran, (eds),
*Diagrammatic Reasoning: Cognitive and Computational Perspectives*, pages 209-232. Cambridge, MA: AAAI Press/The MIT Press. - Barwise, J., and Shimojima, A., 1995,
"Surrogate Reasoning",
*Cognitive Studies: Bulletin of Japanese Cognitive Science Society*, 4(2): 7-27. - Block, N., (ed.), 1981,
*Imagery*. Cambridge, MA: MIT Press. - -----, 1983, "Mental pictures and cognitive
science",
*The Philosophical Review*, 92: 499-541 - Booch, G., Rumbaugh, J., and Jacobson, I., 1998,
*The Unified Modeling Language User Guide*. Addison-Wesley. - Carroll, L., 1896,
*Symbolic Logic*. Dover. - Chandrasekaran, B., Glasgow, J., and Narayanan,
N. Hari, (eds.), 1995,
*Diagrammatic Reasoning: Cognitive and Computational Perspectives*. Cambridge, MA: AAAI Press/The MIT Press. - Cummins, R., 1996,
*Representations, Targets, and Attitudes*. Cambridge, MA: MIT Press. - Dennett, D., 1981, "The nature of images and the introspective trap", in Block 1981, pages 87-107.
- Englebretsen, G.,, 1992, "Linear Diagrams
for Syllogisms (with Relationals)",
*Notre Dame Journal of Formal Logic*, 33(1): 37-69. - Euler, L., 1768,
*Lettres à une Princesse d’Allemagne*. St. Petersburg; l’Academie Imperiale des Sciences. - Fodor, J., 1981, "Imagistic representation", in Block 1981, pages 63-86.
- Frege, G., 1879,
*Begriffsschrift: eine der arithmetischen nachgebildete Formelsprache des reinen Denkens*, Halle am See: Louis Nebert - Gardner, M., 1958,
*Logic Machines and Diagrams*. Sussex: Harvester Press. - Goodman, N., 1968,
*Languages of Art: an approach to a theory of symbols*. London: Oxford University Press. - Grigni, M., Papadias, D., and Papadimitriou,
C., 1995, "Topological Inference", in
*International Joint Conference on Artificial Intelligence (IJCAI ’95)*, pages 901-907, Cambridge, MA: AAAI Press. - Gurr, C., Lee, J., and Stenning, K., 1998,
"Theories of diagrammatic reasoning: Distinguishing component
problems",
*Minds and Machines*, 8: 533-557. - Hammer, E., 1995, "Reasoning with Sentences and
Diagrams",
*Notre Dame Journal of Formal Logic*, 35(1): 73-87 - Hammer, E., and Shin, S., 1998,
"Euler’s Visual Logic",
*History and Philosophy of Logic*, 19: 1-29. - Harel, D., 1988, "On Visual Formalisms",
*Communications of the ACM*, 31(5): 514-530. - Howell, R., 1976, "Ordinary Pictures, Mental
Representations, and Logical Forms",
*Synthese*, 33: 149-174. - Jamnik, M., Bundy, A., and Green, I., 1999,
"On Automating Diagrammatic Proofs of Arithmentic Arguments",
*Journal of Logic, Language, and Information*, 8(3): 297-321. - Kauffman, L. 1991,
*Knots and Physics*. World Scientific, Singapore. - Kosslyn, S., 1980,
*Image and Mind*. Cambridge, MA: Harvard University Press. - -----, 1994,
*Image and Brain: the resolution of the imagery debate*. Cambridge, MA: MIT Press. - Lambert, J. H., 1764,
*Neues Organon*, Berlin: Akademie Verlag, 1990. - Larkin, J., and Simon, H., 1987, "Why a
Diagram is (Sometimes) Worth 10,000 Words",
*Cognitive Science*, 11: 65-99. - Lemon, O., de Rijke, M., and Shimojima, A.,
1999, "Efficacy of Diagrammatic Reasoning" (Editorial),
*Journal of Logic, Language, and Information*, 8(3): 265-271. - -----, 1997b, Spatial Logic and
the Complexity of Diagrammatic Reasoning.
*Machine GRAPHICS and VISION*, 6(1): 89-108, 1997. (Special Issue on Diagrammatic Representation and Reasoning). - -----, 1998, "On the insufficiency of linear
diagrams for syllogisms",
*Notre Dame Journal of Formal Logic*, 39(4): 573-580. - Malinas, G., 1991, "A Semantics for
Pictures",
*Canadian Journal of Philosophy*, 21:3, pages 275-298. - Narayanan, N., 1993, "Taking issue/forum:
The imagery debate revisited",
*Computational Intelligence*, 9(4): 303-435. - Peacocke, C., 1987, "Depiction",
*The Philosophical Review*, 96: 383-410 - Peirce, C.S., 1933,
*Collected Papers*. Cambridge, MA: Harvard University Press. - Pylyshyn, Z., 1981, "Imagery and Artificial
Intelligence", in N. Block, (ed.),
*Readings in Philosophy of Psychology*, volume 2, pages 170 -196. Cambridge, MA: Harvard University Press. - Roberts, D., 1973,
*The Existential Graphs of Charles S. Peirce*. The Hague: Mouton. - Russell, B., 1923, "Vagueness", in J. Slater,
(ed.),
*Essays on Language, Mind, and Matter: 1919-26*, The Collected Papers of Bertrand Russell, pages 145-154. London: Unwin Hyman. - Shepard, R., and Metzler, J., 1971, "Mental
rotation of three-dimensional objects",
*Science*, (171): 701-3. - Shimojima, A., 1996a,
*On the Efficacy of Representation*. Ph.D. thesis, Indiana University. - -----, 1999, "Constraint-Preserving
Representations", in L. Moss, J. Ginzburg, and M. de Rijke, (eds),
*Logic, Language and Computation: Volume 2*, CSLI Lecture Notes #96, pages 296-317. Stanford: CSLI Publications. - Shin, S., 1994,
*The Logical Status of Diagrams*. Cambridge: Cambridge University Press. - Sloman, A., 1971, "Interaction between
philosophy and ai: The role of intuition and non-logical reasoning in
intelligence", in
*Proceedings Second International Joint Conference on Artificial Intelligence*, Morgan Kaufmann. - -----, 1985, "Why we need many knowledge
representation formalisms", in M. Bramer, (ed.),
*Research and Development in Expert Systems*, pages 163-183. - -----, 1995, "Musings on the roles of logical and nonlogical representations in intelligence", in [Chandrasekaran et al., 1995], pages 7-32.
- Sober, E., 1976, "Mental Representations",
*Synthese*, 33: 101-148 - Stenning, K., 1999, "Review of
*Das Spiel der Logik*, by Lewis Carrol",*Journal of Symbolic Logic*, 64: 1368-1370. - Stenning, K., and Lemon, O., 2001, "Aligning
Logical and Psychological Perspectives on Diagrammatic Reasoning",
*Artificial Intelligence Review*, 15(1-2): 29-62. (Reprinted in*Thinking with Diagrams*, Kluwer, 2001.) - Tye, M., 1991,
*The Imagery Debate*, Cambridge, MA: MIT Press. - Venn, J., 1881,
*Symbolic Logic*, London: Macmillan. - Wang, D., and Lee, J., 1993, "Visual
Reasoning: its Formal Semantics and Applications",
*Journal of Visual Languages and Computing*, 4: 327-356. - Wittgenstein, L., 1921,
*Tractatus Logico-Philosophicus*, B. Pears and B. McGuinness (trans), London: Routledge & Kegan Paul, 1961 - Zeman, J., 1964,
*The Graphical Logic of C. S. Peirce*. Ph.D. thesis, University of Chicago.

- Barwise, J., and Hammer, E., 1994,
"Diagrams and the Concept of a Logical System", in D. Gabbay, (ed),
*What is a Logical System?*New York: Oxford University Press. - Hammer, E., 1998, "Semantics for Existential
Graphs",
*Journal of Philosophical Logic*, 27: 489-503 - -----, 1995,
*Logic and Visual Information*. Studies in Logic, Language, and Computation. Stanford: CSLI Publications and FoLLI. - Hammer, E., and Shin, S., 1996, "Euler
and the Role of Visualization in Logic", in J. Seligman and
D. Westerståhl, (eds),
*Logic, Language and Computation: Volume 1*, CSLI Lecture Notes #58, pages 271-286. Stanford: CSLI Publications. - Kneale, W., and Kneale, M., 1962,
*The Development of Logic*. Oxford: Clarendon Press - Lemon, O., 1997, "Review of
*Logic and Visual Information*, by E. M. Hammer",*Journal of Logic, Language, and Information*, 6(2): 213-216. - Lemon, O., 2001, "Comparing the Efficacy of
Visual Languages", in D. Barker-Plummer, D. Beaver, P. Scotto di
Luzio, and J. van Benthem, (eds),
*Logic, Language, and Diagrams*. Stanford: CSLI Publications (in press). - Roberts, D., 1992, "The Existential Graphs of
Charles S. Peirce",
*Computer and Math. Applic.*, (23): 639-663. - Shimojima, A., 1996b, "Operational constraints in
diagrammatic reasoning", in J. Barwise and G. Allwein, (eds),
*Logical Reasoning with Diagrams*, New York: Oxford University Press, pages 27-48 - -----, 1996c, "Reasoning with Diagrams and
Geometrical Constraints", in J. Seligman and D. Westerståhl,
(eds),
*Logic, Language and Computation: Volume 1*, CSLI Lecture Notes #58, pages 527-540. Stanford, CSLI Publications. - Shin, S., 1991, "A Situation-Theoretic
Account of Valid Reasoning with Venn Diagrams", in J. Barwise,
J. Gawron, G. Plotkin, and S. Tutiya, (eds),
*Situation Theory and its Applications: Volume 2*, CSLI Lecture Notes #26, pages 581-605. Stanford: CSLI Publications. - -----, 1999, "Reconstituting Beta Graphs
into an Efficacious System",
*Journal of Logic, Language, and Information*, 8: 273-295. - -----, 2000, "Reviving the Iconicity of Beta
Graphs", in Anderson, Cheng, and Haarslev, (eds),
*Theory and Application of Diagrams*, pages 58-73. Springer-Verlag - -----, 2001a, "Multiple Readings of
Peirce’s Alpha Graphs", in M. Anderson and B. Meyer, (eds),
*Thinking with Diagrams*. Dordrecht: Kluwer. (in press). - -----, 2001b,
*The Iconic Logic of Peirce’s Graphs*. Cambridge, MA: MIT Press (in press). - Sowa, J., 1984,
*Conceptual Structures: Information Processing on Mind and Machine*. London: Addison Wesley. - -----, 2000,
*Knowledge Representation: Logical, Philosophical, Computational Foundations*. Belmont, CA: Brooks/Cole. - Stenning, K., and Oberlander, J., 1995, "A
Cognitive Theory of Graphical and Linguistic Reasoning: Logic and
Implementation",
*Cognitive Science*, 19(1): 97-140. - Tufte, E., 1983,
*The Visual Display of Quantitative Information*, Connecticut, Graphics Press. - -----, 1990,
*Envisioning Information*, Connecticut, Graphics Press.

- Diagrammatic Reasoning site (University of Hartford)
- Visual Inference Lab (Indiana University)
- Feynman diagrams (Stanford University)
- Diagrammatics
- Knot Theory (York University)

and

*First published: August 28, 2001*

*Content last modified: August 28, 2001*