Eliasmith, C. (1997) *Structure
without symbols: Providing a distributed account of high-level
cognition*. Southern Society for Philosophy and Psychology.
Conference March, 1997.

*Philosophy-Neuroscience-Psychology Program,
Department of Philosophy, Washington University in St. Louis,
Campus Box 1073, One Brookings Drive, St. Louis, MO 63130-4899,
chris@twinearth.wustl.edu*

There has been a long-standing debate between symbolicists and connectionists concerning the nature of representation used by human cognizers. In general, symbolicist commitments have allowed them to provide superior models of high-level cognitive function. In contrast, connectionist distributed representations are preferred for providing a description of low-level cognition. The development of Holographic Reduced Representations (HRRs) has opened the possibility of one representational medium unifying both low-level and high-level descriptions of cognition. This paper describes the relative strengths and weaknesses of symbolic and distributed representations. HRRs are shown to capture the important strengths of both types of representation. These properties of HRRs allow a rebuttal of Fodor and McLaughlin's (1990) criticism that distributed representations are not adequately structure sensitive to provide a full account of human cognition.

The earliest computational cognitive models
were typically symbolic (see Newell, 1990). Commonly, such models
employed logic-like rules and manipulated discrete symbols to
mimic human cognitive capacities. This *symbolicist* tradition
continues, and has had much success modeling high-level cognitive
tasks such as planning, skill acquisition, analogical reasoning,
problem solving, logical reasoning, elementary sentence verification
and verbal recognition and recall (Newell, 1990; Falkenhainer
*et.* *al.*, 1989).

More recently, *connectionism* has provided
another means of modeling cognitive function. These models can
be distinguished from symbolicist models on the basis of a commitment
to a different type of representation. Where symbolicists employ
discrete symbolic representation, connectionists are committed
to a distributed form of representation**1**. The greatest successes of connectionist models have
been on low-level cognitive tasks such as categorization, recognition,
and simple learning (Rumelhart and McClelland, 1986; Churchland,
1989).

Critics of connectionism often contend that
these models, though impressive in some ways, have little to do
with high-level human cognition. The models typically lack the
right properties; namely, those exhibited by users of natural
language (Fodor and Pylyshyn, 1988; Fodor and McLaughlin, 1990;
*c. f.* Bechtel, 1995). Conversely, critics of symbolicism
claim that such models do not seriously consider the types of
processing taking place in the brain. The result is the inability
of symbolicist models to account for developmental phenomena,
the plasticity of the brain, and many low-level cognitive tasks
(Churchland and Sejnowski, 1992; Churchland, 1989).

Until recently, connectionist distributed representations have been unable to provide an effective means of capturing the complex embedded structure necessary for many high-level cognitive tasks. Though there is general agreement among both connectionists and neuroscientists that representation in the brain is distributed (see footnote 1), there has been no way of providing distributed models of structure-dependent tasks such as language processing, planning, or analogy (Hinton, 1986; Miikkulainen and Dyer, 1988; Smolensky, 1988; Churchland, 1989; Churchland and Sejnowski, 1992). Fodor and McLauglin (1990) provide a general critique of distributed models inspired by these traditional difficulties of connectionism. They intend to show that distributed representations are not amenable to high-level cognition.

In order for us to provide models which can explain phenomena from the lowest to the highest levels of human cognition its seems we have two choices:

- Provide symbolic representations
which are flexible
**2**or; - Provide distributed representations which can be structure sensitive.

There are a number of reasons why distributed
representations are more likely to provide a unifying representational
medium than symbolic representations. First, distributed representations
can be learned from the environment of the cognizer (Hinton, 1986).
Second, distributed representations are equally applicable to
all modes of perception (*i.e.* vision, hearing, olfaction,
*etc*.). Third, there have been recent developments in distributed
representations that provide an effect means of capturing embedded
structure (Smolensky, 1990; Plate, 1993). And fourth, distributed
representations are undeniably present in the brain and provide
a degree of neurological realism not found in symbolic models
(Churchland and Sejnowski, 1992; Churchland, 1995).

In this paper I will develop a version of the
second option. Section 2 presents a general explanation of distributed
representation. Section 3 describes *holographic reduced representations*
(HRRs) which are a form of distributed representations that effectively
capture structure. Finally, section 4 discusses how these representations
overcome the criticisms of Fodor and McLaughlin (1990) of connectionist
distributed representations. Fodor and McLaughlin's failure is
an indication of the ability of such representations to an play
important role in both low-level and high-level cognitive modeling.

The concept of *distributed representation
*is a product of joint developments in the neurosciences and
in connectionist work on recognition tasks (Churchland and Sejnowski,
1992). To introduce distributed representations as employed by
connectionists, it is helpful to contrast them with symbolic representations
of symbolicism.

A symbol, such as a word or a number, is an everyday occurrence for most of us. Of course, symbols are not restricted to being words or numbers, but this is by far their most common guise in symbolicist models. From computational, psychological, and neurological standpoints, there are a number of shortcomings in using purely symbolic representation in modeling human behavior. I will briefly examine three of the more important limitations of symbolic representation that serve to distinguish them from distributed representations.

First, symbolic representations are strongly propositional. Thus, when this method of representation is used in non-language based tasks such as image processing it becomes extremely difficult to explain many psychological findings (Kosslyn, 1994). Similarly, taste, sound, touch, and smell are awkward to handle with symbolic representations. In contrast, distributed representations are equally well suited to all modes of perception.

Second, symbolic representations are 'all-or-none';
this property is referred to as *brittleness*. There is no
such thing as the degradation of a symbolic representation, it
is either there (whole), or it is not there (broken). The brittleness
of symbolic representations is highly psychologically unrealistic.
People often exhibit behavior that indicates partial retrieval
of representations, as exemplified by 'tip-of-the-brain' recall,
prosopagnosia (loss of face recognition), and image completion.
Furthermore, minor damage to a symbolic conceptual network causes
loss of entire concepts, whereas a distributed network loses accuracy,
as people do, not entire concepts (Churchland and Sejnowski, 1992).

Third, symbolic representations are not, in
themselves, statistically sensitive (Smolensky, 1995). In other
words, they are not constructed based on statistical regularities
found in the environment. Therefore, symbolic representations
are not amenable to modeling low-level perception. This sort of
low-level perceptual learning seems to be common to all animals,
and is an important part of human development. It is the case,
however, that though symbolic representations are not statistically
sensitive, they are superbly structurally sensitive. To many,
this had seemed a reasonable trade-off in light of structurally
insensitive alternatives. However, with the more recent development
of distributed representations that exhibit both structural *and*
statistical sensitivity, this trade-off is no longer easily justifiable.

Because of the limitations of *symbolic*
representations, many cognitive scientists have searched for,
and found an alternative; *distributed* representations.
Distributed representations are considerably less common in everyday
life. As an explanatory aid, consider the following analogy (to
which we will return in section 3): imagine all of your concepts
are mapped to the surface of a sphere - a conceptual golf ball,
if you will (figure 1). Each concept is nestled in a dimple on
surface of the golf ball. The more similar two concepts are, the
closer together they will be on the surface of the golf ball.
Also, the distance from the center of the golf ball to any particular
dimple (*i.e.* concept) is approximately the same (one golf
ball radius).

The easiest way to identify the position of
a concept on the surface of the ball is to provide its coordinates.
These coordinates are in the units of 'golf ball radii' and are
continuous real values. Thus, in figure 1, 'girl' has coordinates
(0.5, 0.5, 0.707). Such a set of coordinates is commonly referred
to as a *vector*. This representation of 'girl' is a *distributed
representation*, because the 'girl' dimple can only be located
through knowing *all three* values. Thus, the concept of
'girl' is shared, or distributed, across the three dimensions
of the golf ball's coordinate system.

Notably, a dimple on a golf ball is not a single point in space, rather it is a collection of points which are in proximity on the surface of a sphere. This makes the dimple particularly appealing as an analogy for our concepts for the following reasons:

- A
*prototype*for a concept would be the center of the dimple. - Multiple vectors define the dimple/concept's boundaries; thus it is a collection of examples. Typically, these 'boundary' concepts would be specific examples which the cognizer had observed from its environment.
- Because of 2, the prototype for a concept would not be defined by one particular example, but rather a 'superposition', or 'adding together' of all available examples of the concept.

We will revisit the conceptual golf ball in our examination of the particular type of distributed representations which allow us to successfully capture structure (section 3).

Earlier, I noted that much of the reason researchers were motivated to look for alternate forms of representation was due to the failings of symbolic representations. In particular, the propositional, brittle, and statistically insensitive nature of symbolic representations were shown to be problematic. Distributed representations do not fall prey to these short-comings. Rather, distributed representations:

- Have been successfully applied to visual (Qian and Sejnowski, 1988; Raeburn, 1993), olfactory (Skarda and Freeman, 1987), auditory (Lazzaro and Mead, 1989) and tactile problems;
- Have been proved to degrade gracefully with
noise (Churchland, 1992) and are commonly tested with simulated
lesions (
*i.e.*a removal of part of the representation). - Are the natural result of organization of statistical input and thus provide a natural means to capturing semantic information (Smolensky, 1995; Sarle, 1994);

Furthermore, distributed representations have a number of properties which lend significant psychological and neurological plausibility to models employing them. Most often, distributed representations:

- Represent concepts continuously;
- Are processed in parallel, and;
- Can be learned using proven methods (Hinton,
1986; Gorman and Sejnowski, 1988; Miikkulainen and Dyer, 1988;
Le Cun, Boser
*et. al*., 1990; Plate, 1993).

These powerful properties, coupled with the ability of distributed representations to avoid short-comings of symbolic representation, provide a solid foundation for realistic computational models of human cognition. Armed with the strengths of distributed representations and our conceptual golf ball analogy, we will now examine the type of representations that can provide structure without symbols.

*Holographic reduced representations* or *HRRs* are structurally sensitive,
but distributed vector representations (Plate, 1994). These representations
are constructed, or *encoded*, using a form of vector multiplication
called *circular convolution* and are related to the better-known
*tensor products* of Smolensky (Smolensky, 1990). Decoding
of HRRs is performed using the approximate inverse of circular
convolution, an operation called *correlation* (see appendix
A for the algebraic details of these operations).

The HRR representations are "holographic"
because the encoding and decoding operations (*i.e.* convolution
and correlation) used to manipulate these complex distributed
representations are the same as those which underlie explanations
of holography (Borsellino and Poggio, 1973). The convolution of
two HRRs creates a third unique HRR which encodes the information
present in the two previous HRRs. Importantly, this new HRR is
*not* similar to either of its components, though the components
may be retrieved through decoding the new HRR with the correlation
operator. These operations are easiest to understand through a
simple illustration. Let A, B and C be distributed HRR vectors
(*i.e.* as per the concept 'girl' presented in section 2).
If C = A ƒ* B (read: C equals A convolved with B) then, C # A
B (read: C correlated with A approximately equals B) and C # B
A (see figure 2).

*Figure 2*.
HRR operations depicted on the conceptual golf ball. Vector C
is the circular convolution of vectors A and B. Vector D is the
normalized superposition of A and B.

Along with convolution another operation, *superposition*,
can be used to combine two HRR vectors. Superposition of two vectors
simply results in their sum, and is written: D = A + B (see figure
2). Superimposing two vectors results in a vector which is very
similar to the original two vectors but cannot provide a perfect
reconstruction of the original vectors. This contrasts with convolution,
in which the resulting vector is nothing like the original two
vectors; in fact, the expected similarity of either of the original
vectors to their convolution is zero (Plate, 1994, p. 57). In
both cases, some of the information in the original HRR vectors
is lost in their combination. Hence, HRRs are considered 'reduced'
representations. Upon encoding a new representation from a number
of others (*e.g.* constructing the representation for a particular
'girl' from 'female', 'small', 'human', *etc*.), the new
representation does not contain all of the information present
prior to encoding. In other words, the new representation is *noisy*.
Nevertheless, these representations are extremely effective at
capturing complex relational information. Furthermore, the noise
occurring in HRRs seems to have a neurological counterpart (Andersen,
personal comment).

When decoding an HRR vector the resultant vector
must be recognized even though it is not expected to be identical
to the vector that was encoded. The process of recognizing a vector
is accomplished through use of another operator called the dot
product and represented as 'ï'. The dot product of two vectors
is the sum of the product of each of the corresponding elements
of the vectors. For normalized vectors, the resulting scalar is
equivalent to the length of one of the vectors projected on to
the other. This relative length value can be used as a measure
of the vectors' similarity. Because all of our conceptual vectors
are normalized to one golf ball radius, we can use the dot product
operation to determine the similarity of *any* two vectors.

A number of properties of HRRs make them promising candidates for modeling human cognition. First, HRRs are distributed representations. This means that they have all of the benefits associated with distributed representations. Therefore, HRRs (see section 2): 1. Are sensitive to statistical input; 2. Provide cross-modal representations; 3. Degrade gracefully; 4. Represent concepts continuously; 5. Can be processed in parallel, and; 6. Are subject to the many learning algorithms already developed.

Second, HRRs accommodate arbitrary variable binding through the use of convolution. Third, HRRs can effectively capture embedded structure (Plate, 1994; Eliasmith and Thagard, forthcoming). Fourth, unlike tensor products, and most other distributed representations which use vector multiplication, HRRs are fixed dimension vectors. Thus, convolving two three-dimensional vectors results in another three-dimensional vector - not a six- or nine-dimensional vector. Consequently, HRRs are not subject to an explosion in the size of representations as the structures represented become more complex. This property also allows HRRs of various structural depths to be easily comparable to each other with out "padding" the representation, as is necessary with tensor products. Fifth and finally, convolution can be implemented by a recurrent connectionist network (Plate, 1993). The potential for implementation in a recurrent network supports the neurological plausibility of HRRs. Though the extent of neurological realism of any such artificial neural networks may be disputed, it is indisputable that they are more neurologically realistic than either localist connectionist or symbolic models (Smolensky, 1995).

Most important to modeling high-level cognition are the structural properties. The ability of HRRs to accommodate arbitrary variable bindings and embedded structure will allow distributed models to handle structures as complex those in symbolicist systems. As well, HRRs maintain the traditional strengths of distributed representations and can be applied to modeling low-level cognition.

Providing a form of distributed representation
which can be used to model higher cognitive function provides
support to the contentious claim that connectionist ideas will
scale to the complexity of high-level human cognition; a feat
some thought may not be possible (Gentner and Markman, 1993).
More particularly, the claims by Fodor and Pylyshyn (1988) and
more recently Fodor and McLaughlin (1990) that connectionist systems
*can not* exhibit certain characteristics necessary for modeling
human cognition can be directly challenged.

For Fodor and McLaughlin, the most obvious
failing of distributed representations, including tensor products
(and thus presumably HRRs), is a lack of *systematicity*.
They characterize systematicity as follows (Fodor and McLaughlin,
1990, p. 184):

[A]s a matter of psychological law, an organism is able to be in one of the states belonging to the family [of semantically related mental states] only if it is able to be in many of the others…[for example] You don't find organisms that can think the thought that the girl loves John but can't think the thought that John loves the girl.

A system employing HRRs can satisfy this definition of systematicity. In representing structure, an HRR representation consists in a series of convolutions and superpositions. For example, to represent the sentence 'John loves the girl' we would perform the following operations:

S1 = relation*love + agent*John + patient*girl

Where each of 'relation', 'love', 'agent',
'john', 'patient', and 'girl' are HRRs. The resulting sentence
vector, S1, will successfully encode the example proposition proposed
by Fodor and McLauglin. Furthermore, constructing the complementary
sentence 'The girl loves John' uses exactly those HRRs needed
for S1, *i.e.*:

S2 = relation*love + agent*girl + patient*John

Thus, an HRR system capable of encoding the
first sentence (S1) can *necessarily* encode the second (S2).
In other words, an HRR system is systematic in just the right
way. Also note that vectors S1 and S2, if compared to one another
would be quite similar, as expected, but distinct nonetheless.
If sentence S1 is queried as to the agent of the sentence (by
performing the operation S1#agent), the result indicates that
the agent is 'John'. Conversely, if S2 is queried as to the agent
(S2#agent), the result is the vector for 'girl'.

Accounting for systematicity with such little
effort naturally raises the concern that this sort of HRR system
is simply a new implementation of the 'classical' (*i.e.*
Fodor and McLaughlin's) solution. This, however, is clearly not
the case. Fodor and McLaughlin provide the following definition
of a 'classical constituent' (ibid., p. 186):

[F]or a pair of expression types E1, E2, the
first is a *Classical* constituent of the second *only
if* the first is tokened whenever the second is tokened.

With HRRs it is not the case that when the sentence S1 is tokened, any of the components of that sentence are tokened. We can easily combine sentences S1 and S2 to form the following sentence, S3:

S3 = relation*cause + agent*S1 + patient *S2

In tokening S1 and S2 to form this sentence,
we *in no way* tokened the vector 'John'. Rather, a vector
which, *when operated upon by a given operator*, produces
the vector 'John' was tokened. Thus, constituents of S1 and S2
are 'non-classical' in the sense provided by Fodor and McLaughlin.

Fodor and McLaughlin would likely agree, but they would conclude that therefore, HRRs are not systematic. In the abstract of their 1990 paper, they claim to show (ibid., p. 183, my italics):

[Such] representations fail to explain systematicity
*because* they fail to exhibit the sort of constituents that
can provide domains for structure sensitive mental processes.

While it is true that the constituents of HRR
representations are non-classical, it is not true that this sort
of constituent is necessary for providing structure sensitive
processes. Eliasmith and Thagard (forthcoming) provide a detailed
model of analogical reasoning in which HRRs are used as the representational
medium. This model, called *Drama*, is as structurally sensitive
as any competing symbolic models (see Falkenhainer *et.*
*al.,* 1989; Mitchell, 1993). The relations between and within
sentences describing given analogies is correctly captured with
HRRs. The domain of human analogy making is one of the most structurally
demanding of those tackled by computational models of cognition
(Gentner and Markman, 1993). If it were truly the case, as Fodor
and McLauglin claim, that representational systems employing HRRs
were necessarily structurally insensitive, it would be impossible
to provide a model of analogy. There can be little stronger proof
that HRR systems can indeed be systematic than the existence of
highly structure sensitive HRR models like *Drama*.

In sum, HRR systems meet Fodor and McLaughlin's
(1990) criteria for being systematic. However, HRR systems do
not meet their criteria for using classical constituents and are
thus not an implementation of the classical solution to the systematicity
problem. Therefore, the structural sensitivity necessary for systematicity
does not have to be provided by symbolic models, as is demonstrated
in the *Drama* model of analogy.

Holographic reduced representations (HRRs) provide an important new means of bridging the gap between low-level and high-level cognitive models. Because HRRs are distributed representations, they have all of the properties of this form of representation which make them ideal for modeling low-level cognitive processes. With the four operators of convolution, correlation, superposition, and the dot product, distributed vectors can be used to effectively capture structural relations. Each of these operators is naturally implemented in artificial neural networks and thus there is no loss of neurological plausibility in their usage (Plate, 1994).

The symbolicist monopoly of high-level cognitive models can be seriously challenged by distributed systems which employ HRRs. The attempt of Fodor and McLaughlin (1990) to show that such representations were unable to capture structure effectively failed in both theory and in practice (Eliasmith and Thagard, forthcoming). The potential of HRRs remains to be seen, but by all indications, this representational medium can provide a means of unifying low-level and high-level models of human cognitive processes.

* Thanks to Paul Thagard, Cameron Shelley, Poco deNada, and to Tony Plate for his insights into the workings of HRRs. This research was supported in part by the Natural Sciences and Engineering Research Council of Canada.

- Many connectionist models employ ‘local’ representations which are very much like symbolicist symbols. However, these sorts of representations are often chosen for pragmatic reasons and are considered non-ideal (Holyoak and Thagard, 1989). There is strong neurological evidence that such representations are not employed by the brain and often distributed models are considered the normative goal for connectionists (Churchland and Sejnowski, 1992; Eliasmith, in press; Sejnowski, 1995).
- Section 2 details the short-comings of symbolic representations which must be addressed to provide the required flexiblity.

Bechtel, W. (1995). Connectionism. __A companion
to the philosophy of mind__. Cambridge, MA, Blackwell.

Borsellino, A. and T. Poggio (1973). 'Convolution
and correlation algebras.' __Kybernetik__ **13**: 113-122.

Churchland, P. (1989). __A neurocomputational
perspective__. Cambridge, MA, MIT Press.

Churchland, P. M. (1995). Machine stereopsis:
A feedforward network for fast stereo vision with movable fusion
plane. __Android Epistemology__. Menlo Park, MIT Press.

Churchland, P. S. and T. Sejnowski (1992).
__The computational brain__. Cambridge, MA, MIT Press.

Clark, A. and J. Toribio (1994). 'Doing without
representing?' __Sythese__ **101**: 401-431.

Eliasmith, C. (in press). "The third contender:
A critical examination of the dynamicist theory of cognition."
__Philosophical Psychology__.

Eliasmith, C. and P. Thagard (forthcoming) "Integrating structure and meaning: A distributed model of analogical mapping."

Falkenhainer, B., K. D. Forbus, and D. Gentner
(1989). 'The structure-mapping engine: Algorithms and examples.'
__Artificial Intelligence__ **41**: 1-63.

Fodor, J. and B. McLaughlin (1990). 'Connectionism
and the problem of systematicity: Why Smolensky's solution doesn't
work.' __Cognition__ **35**: 183-204.

Fodor, J. and Z. Pylyshyn (1988). 'Connectionism
and cognitive architecture: A critical analysis.' __Cognition__
**28**: 3-71.

Gentner, D. and A. B. Markman (1993). Analogy
- watershed or Waterloo? Structural alignment and the development
of connectionist models of analogy. __Advances in neural information
processing systems__. San Mateo, CA, Morgan Kaufmann.

Gorman, R. P. and T. J. Sejnowski (1988). 'Analysis
of hidden units in a layered network trained to classify sonar
targets.' __Neural Networks__ **1**: 75-89.

Harnad, S. (1992). Connecting object to symbol
in modelling cognition. __Connectionism in Context__. London,
Springer-Verlag.

Hinton, G. E. (1986). __Learning distributed
representations of concepts__. Eighth Conference of the Cognitive
Science Society, Lawrence Erlbaum Associates.

Hofstadter, D. (1995). A review of mental leaps:
analogy in creative thought. AI Magazine. **16: **75-80.

Holyoak, K. and P. Thagard (1995). __Mental
leaps: Analogy in creative thought__. Cambridge, MA, MIT Press.

Holyoak, K. J. and P. Thagard (1989). 'Analogical
mapping by constraint satisfaction.' __Cognitive Science__
**13**: 295-355.

Kosslyn, S. (1994). __Image and brain: The
resolution of the imagery debate__. Cambridge, MA, The MIT Press.

Lazzaro, J. and C. Mead (1989). 'A silicon
model of auditory localization.' __Neural Computation__ **1**:
47-57.

Le Cun, Y., B. Boser, J. Denker, D. Henderson,
R. Howard, W. Hubbard, and L. Jackel (1990). Handwritten digit
recognition witha back-propagation network. __Advances in neural
information processing systems__. San Mateo, Morgan Kaufmann.
396-404.

Legendre, G., Y. Miyata, and P. Smolensky (1994).
__Principles for an integrated connectionist/symbolic theory
of higher cogntion__. Hillsdale, NJ, Lawrence Erlbaum Associates.

Miikkulainen, R. and M. G. Dyer (1988). __Encoding
input/output representations in connectionist cognitive systems__.
Connectionists models summer school, San Mateo, Morgan Kaufmann
Publishers.

Mitchell, M. (1993). __Analogy-making as perception__.
Cambridge, Massachusetts, MIT Press.

Plate, T. A. (1993). Holographic recurrent
networks. __Advances in neural information processing systems__.
San Mateo, Morgan Kaufmann. 34-41.

Plate, T. A. (1994). Distributed representations and nested compositional structure. University of Toronto.

Qian, N. and T. J. Sejnowski (1988). __Learning
to solve random-dot seterograms of dense and transparent surfaces
with recurrent backpropagation__. Connectionist Models Summer
School, San Mateo, Morgan Kaufmann Publishers.

Raeburn, P. (1993). Reverse engineering the human brain. Technology Review.

Rumelhart, D., P. Smolensky, G. Hinton, and
J. McClelland (1986). Schemata and sequential thought processes
in PDP models. __Parallel distributed processing: Explorations
in the microstructure of cognition__. Cambridge MA, MIT Press/Bradford
Books. 7-57.

Rumelhart, D. E. and J. L. McClelland, Ed.
(1986). __Parallel distributed processing: Explorations in the
microstructure of cognition__. Cambridge MA, MIT Press/Bradford
Books.

Sarle, W. S. (1994). __Neural networks and
statistical models__. 19th Annual SAS Users Group International
Conference.

Skarda, C. A. and W. J. Freeman (1987). 'How
brains make chaos in order to make sense of the world.' __Behavioral
and Brain Sciences__ **10**: 161-195.

Smolensky, P. (1988). 'On the proper treatment
of connectionism.' __Behavioral and Brain Sciences__ **11**(1):
1-23.

Smolensky, P. (1990). 'Tensor product variable
binding and the representation of symbolic structures in connectionist
systems.' __Artificial Intelligence__ **46**: 159-217.

Smolensky, P. (1995). Computational models
of mind. __A companion to the philosophy of mind__. Cambridge,
MA, Blackwell.

Consider a set *E* of elements which are
holographic reduced representations (HRRs). A member of *E*
is an n-dimensional vector whose contents may represent an image,
a proposition, a concept, *etc*. The *prima facie* similarity
of two vectors is captured by their dot product. The operations
necessary to encode and decode HRRs can be understood as follows:

Let be the space of item vectors in n-dimensions, and let be the space of stored vectors in n-dimensions.

Let

be the encoding operation (circular convolution),

be the decoding operation (circular correlation), and

be the superposition operation (addition).
These three operations make it possible to store any relations
necessary for generating the network of relations amongst elements
of *E*.

More precisely, the circular convolution operation
ƒ
is often referred to simply as convolution and consists of the
following operations for **c** = **a **ƒ **b** where **a**, **b**, and **c** are n-dimensional
vectors:

co = aobo + anb1 + an-1b2 + ... + a1bn

c1 = an-1bo + an-2b1 + ... + anbn

cn = anbo + an-1b1 + ... + aobn

or

for *j=*0 to* n-*1 (subscripts
are modulo-n)

This operation can be represented as:

Similarly, the circular correlation operation
# is often referred to simply as correlation and consists of the
following operations for **d** = **a **# **c**:

do = aoco + a1c1 + ... + ancn

d1 = anco + aoc1 + ... + an-1cn

dn = a1co + anc1 + ... + aocn

or

for *j=*0 to* n-*1 (subscripts
are modulo-n)

This operation can be represented as:

Notably, the correlation of two vectors **a
**# **c
**can be written as **a***** **ƒ **c** where **a***** **is the *approximate inverse
*of **a** which is defined as:

Let

**a** = {ao, a1, ..., an}

then

**a***** **= {ao, an,
..., a1}

Though the *exact inverse*, **a**-1, could be used to decode
**a **ƒ **c** exactly, this process
results in a lower signal-to-noise ratio in the retrieved vectors
in most instances.