Connectionism

Connectionism is an approach in the fields of cognitive science, neuroscience, psychology and philosophy of mind. Connectionism models mental or behavioral phenomena with interconnected networks of simple units. There are many different forms of connectionism, but the most common forms utilize neural network models.

Contents

1 Basic principles 2 Learning 3 Background 4 Connectionist vs. symbolist debate 5 See also 6 References

Basic principles

The central connectionist principle is that mental phenomena can be described by interconnected networks of simple units. The form of the connections and the units can vary from model to model. For example, units in the network could represent neurons and the connections could represent synapses. Another model might make each unit in the network a word, and each connection an indication of semantic similarity.

Most connectionist models include time, i.e. there is a variable which represents time and the network changes over time. Another extremely common aspect of connectionist models is "activation". At any time a unit in the network has an "activation", which is a numerical number indicating the unit's activity. For example, if the units in the model are neurons the activation could represent the probability that the neuron would generate an action potential spike. If the model is a "spreading activation" model then a unit's activation spreads to all the other units connected to it. Time and spreading activation are always features of neural network connectionist models.

Neural networks are by far the dominant form of connectionist model today. A lot of research utilizing neural networks is carried out under the more general name "connectionist". These modern connectionist models adhere to two major principles regarding the mind.

• Any given mental state can be described as a (N)-dimensional vector of numeric activation values over neural units in a network.
• Memory is created by modifying the strength of the connections between neural units. The connection strengths, or "weights", are generally represented as a (N×N)-dimensional matrix.

Learning

Connectionists generally stress the importance of learning in their models. As a result, many sophisticated learning procedures for neural networks have been developed by connectionists. Learning always involves modifying the connection weights. These generally involve mathematical formula to determine the change in weights when given sets of data consisting of activation vectors for some subset of the neural units.

By formalizing learning in such a way connectionists have many tools at their hands. A very common tactic in connectionist learning methods is to incorporate gradient descent over an error surface in a space defined by the weight matrix. All gradient descent learning in connectionist models involves changing each weight by the partial derivative of the error surface with respect to the weight. Backpropagation, first made popular in the 1980s, is probably the most commonly known connectionist gradient descent algorithm today.

Background

The prevailing form of connectionist models today is known as Parallel Distributed Processing (PDP). PDP form became popular in the 1980s with the release of Parallel Distributed Processing: Explorations in the Microstructure of Cognition - Volume 1 (foundations) & Volume 2 (Psychological and Biological Models), by James L. McClelland, David E. Rumelhart , and the PDP Research Group. PDP's roots are the perceptron theories from the 1950s. In spite of the fact that as early as 1952 Friedrich Hayek posited the idea of spontaneous order in the brain arising out of decentralized networks of simple units (neurons), Hayek's work was never cited in the literature of connectionism.

Another form of connectionist models is the Relational Network framework developed by the linguist Sydney Lamb in the 1960s. Relational Networks have only ever been used by linguists.

An earlier and rather different connectionistic view was held by Edward Thorndike, a turn of the century psychologist who studied learning, with his most famous contributions being work on how cats escaped from puzzle boxes, and his formulation of the Law of Effect. His analysis (and its descendants) are peppered with references to associations between stimuli and responses. Though the S-R aspect has today been abandoned by radical behaviorists and cognitive psychologists (including connectionists), it is easy to impose the notion of association and modification of association strength on connectionist models.

Connectionists are generally in agreement that recurrent neural networks (networks wherein connections of the network can form a directed cycle) are more like the human brain than feedforward neural networks (networks with no directed cycles). A lot of recurrent connectionist models incorporate dynamical systems theory as well. Many researchers, such as the connectionist Paul Smolensky (one of the authors of the original PDP books), have argued that the direction connectionist models will take is towards fully continuous, high-dimensional, non-linear dynamic systems approaches.

Connectionist vs. symbolist debate

Many theorists believe that connectionism is in opposition to symbolism. Symbolism is a more specific form of cognitivism which argues that mental activity is computational, i.e. that the mind is essentialy a Turing machine. The essential differences between symbolist and connectionist approaches are the following:

• Symbolists posit symbolic models that do not resemble underlying brain structure at all, whereas connectionists engage in "low level" modelling, trying to ensure that their models resemble neurological structures.
• Symbolists generally focus on the structure of explicit symbols (mental models) and syntactical rules for their internal manipulation, whereas connectionists focus on learning from environmental stimuli and storing this information in a form of connections between neurons.
• Symbolists believe that internal mental activity consists of manipulation of explicit symbols (see Language of thought), whereas connectionists believe that manipulation of explicit symbols are a poor model of mental activity.

See also

• Language of thought
• Cybernetics
• Eliminative materialism
• Behaviorism

References

• Rumelhart, D.E., J.L. McClelland and the PDP Research Group (1986), Parallel Distributed Processing: Explorations in the Microstructure of Cognition. Volume 1: Foundations, Cambridge, MA: MIT Press
• McClelland, J.L., D.E. Rumelhart and the PDP Research Group (1986), Parallel Distributed Processing: Explorations in the Microstructure of Cognition. Volume 2: Psychological and Biological Models, Cambridge, MA: MIT Press
• Jeffrey L. Elman, Elizabeth A. Bates, Mark H. Johnson, Annette Karmiloff-Smith, Domenico Parisi, Kim Plunkett: Rethinking Innateness. A connectionist perspective on development. MIT Press, 1996