Hopfield Networks: From Neurons to Memory

An interactive visualization of Hopfield networks — one of the most elegant ideas in computational neuroscience. This document walks through the theory, the math, and the intuition behind how a network of simple binary units can store and recall patterns from memory.

Try the live demo — draw patterns, train the network, corrupt them with noise, and watch it reconstruct them.

---

Table of Contents

• Hopfield Networks: From Neurons to Memory
  • Table of Contents
  • The Big Idea
  • The Neuron Model
  • Connecting Neurons: The Weight Matrix
  • Learning: The Hebbian Rule
    • Example
  • Recall: How the Network Remembers
  • The Energy Function
  • Why It Works: Energy Minimization
  • Capacity: How Much Can It Remember?
    • The Overlap Parameter
  • Failure Modes: Spurious States
    • 1. Inverted Patterns
    • 2. Mixture States
    • 3. Spin Glass States
  • Connection to Physics
    • Workflow
    • Keyboard Shortcuts
  • References

---

The Big Idea

Imagine you see half of a friend's face. Instantly, your brain fills in the rest. You don't compute anything consciously — the complete image just appears. Hopfield networks are a mathematical model of this process: content-addressable memory.

Unlike a hard drive where you look up data by address (give me byte 0x4F2A), a Hopfield network retrieves data by content (give me the pattern that looks most like this). You feed in a corrupted, partial, or noisy version of something it has seen before, and it reconstructs the original.

The key insight, due to John Hopfield (1982), is that this can be understood as an energy minimization problem. The stored patterns are valleys in an energy landscape. Recall is the process of rolling downhill into the nearest valley.

---

The Neuron Model

The network consists of $N$ neurons. Each neuron $i$ has a binary state:

$$s_i \in {-1, +1}$$

That's it. No continuous activations, no layers, no backpropagation. Each neuron is either ON ($-1$) or OFF ($+1$).

In this visualization, $N = 400$ neurons are arranged on a 20 x 20 grid. Cyan cells are ON ($-1$), dark cells are OFF ($+1$). The choice of $-1$ for ON is a convention from physics (spin down = active).

The full state of the network at any moment is a vector:

$$\mathbf{s} = (s_1, s_2, \ldots, s_N)$$

This vector lives in a space with $2^N$ possible configurations. For our 400-neuron network, that's $2^{400} \approx 10^{120}$ possible states — more than the number of atoms in the observable universe. The network needs to find a few "good" states (the stored patterns) in this astronomically large space.

---

Connecting Neurons: The Weight Matrix

Every neuron is connected to every other neuron (but not to itself) through a symmetric weight $W_{ij}$:

$$W_{ij} = W_{ji}, \quad W_{ii} = 0$$

The weight $W_{ij}$ represents how strongly neuron $i$ influences neuron $j$. If $W_{ij} > 0$, the neurons excite each other — they want to be in the same state. If $W_{ij} < 0$, they inhibit each other — they want to be in opposite states.

The full weight matrix $\mathbf{W}$ is an $N \times N$ symmetric matrix with zeros on the diagonal. This matrix is the network's memory. Everything the network knows is encoded in these numbers.

In the visualization, the "Per Neuron" weight map shows one row of this matrix reshaped back onto the 20 x 20 grid. Cyan = excitatory connection, red = inhibitory. The "Full Matrix" view shows all $N \times N$ entries as a heatmap.

---

Learning: The Hebbian Rule

How do we choose the weights so the network remembers a set of patterns? We use Hebb's rule, inspired by the neuroscience principle: neurons that fire together, wire together.

Suppose we want to store $P$ patterns, where each pattern $\boldsymbol{\xi}^\mu$ is a vector of $\pm 1$ values:

$$\boldsymbol{\xi}^\mu = (\xi_1^\mu, \xi_2^\mu, \ldots, \xi_N^\mu), \quad \mu = 1, \ldots, P$$

The Hebbian learning rule sets each weight to:

$$\boxed{W_{ij} = \frac{1}{N} \sum_{\mu=1}^{P} \xi_i^\mu , \xi_j^\mu}$$

Let's unpack what this does:

• If neurons $i$ and $j$ are both ON or both OFF in a stored pattern ($\xi_i^\mu = \xi_j^\mu$), their product is $+1$, contributing a positive (excitatory) weight.
• If they disagree ($\xi_i^\mu \neq \xi_j^\mu$), the product is $-1$, contributing a negative (inhibitory) weight.
• The sum over all patterns means the weight reflects the net agreement across all stored memories.
• The $1/N$ factor is a normalization to keep the weights from growing with network size.

This can be written compactly in matrix form:

$$\mathbf{W} = \frac{1}{N} \sum_{\mu=1}^{P} \boldsymbol{\xi}^\mu (\boldsymbol{\xi}^\mu)^\top - \frac{P}{N}\mathbf{I}$$

The $-\frac{P}{N}\mathbf{I}$ term enforces the zero-diagonal constraint ($W_{ii} = 0$), which prevents neurons from reinforcing themselves.

Example

Consider two neurons that are both ON in patterns 1 and 3, but disagree in pattern 2:

Pattern	Neuron $i$	Neuron $j$	Product $\xi_i \xi_j$
1	$-1$ (ON)	$-1$ (ON)	$+1$
2	$-1$ (ON)	$+1$ (OFF)	$-1$
3	$+1$ (OFF)	$+1$ (OFF)	$+1$

$$W_{ij} = \frac{1}{N}(+1 - 1 + 1) = \frac{+1}{N}$$

The net weight is positive — these neurons mostly agree, so they have an excitatory connection.

---

Recall: How the Network Remembers

Recall is an iterative process. We start from some initial state (possibly a corrupted pattern) and repeatedly update neurons until the network settles.

The update rule for neuron $i$:

$$\boxed{s_i \leftarrow \text{sign}\left(\sum_{j=1}^{N} W_{ij} , s_j\right)}$$

In detail:

1. Pick a random neuron $i$
2. Compute its local field (the weighted sum of all other neurons' states): $$h_i = \sum_{j=1}^{N} W_{ij} , s_j$$
3. Update the neuron: $$s_i = \begin{cases} +1 & \text{if } h_i \geq 0 \ -1 & \text{if } h_i < 0 \end{cases}$$
4. Repeat until no neurons change (convergence)

The local field $h_i$ is the "vote" from all other neurons. Neurons connected by positive weights vote for $i$ to match their state. Neurons connected by negative weights vote for $i$ to be opposite. The neuron takes the majority vote.

This is asynchronous updating — one neuron at a time, chosen randomly. This is crucial. Synchronous updating (all at once) can lead to oscillations and doesn't guarantee convergence.

In the visualization, the inference panel shows this computation for a selected neuron: the excitatory sum (positive votes), inhibitory sum (negative votes), the local field $h_i$ (net vote), and whether the neuron would flip. The bar chart shows every individual contribution $W_{ij} s_j$.

---

The Energy Function

Here is where Hopfield's insight connects neural networks to physics. Define an energy function over the network's state:

$$\boxed{E(\mathbf{s}) = -\frac{1}{2}\sum_{i=1}^{N}\sum_{j=1}^{N} W_{ij} , s_i , s_j = -\sum_{i<j} W_{ij} , s_i , s_j}$$

This is a scalar value that depends on the current configuration of all neurons. Lower energy = more "comfortable" configuration.

Key theorem: When a single neuron updates according to the rule above, the energy never increases:

$$\Delta E \leq 0$$

Proof sketch: When neuron $i$ updates, the change in energy is:

$$\Delta E = -\Delta s_i \cdot h_i$$

where $\Delta s_i = s_i^{\text{new}} - s_i^{\text{old}}$ and $h_i = \sum_j W_{ij} s_j$.

If the neuron flips from $-1$ to $+1$ (i.e., $\Delta s_i = +2$), it's because $h_i \geq 0$, so $\Delta E = -2h_i \leq 0$.

If it flips from $+1$ to $-1$ ($\Delta s_i = -2$), it's because $h_i &lt; 0$, so $\Delta E = +2h_i &lt; 0$.

If it doesn't flip, $\Delta E = 0$.

In all cases, $\Delta E \leq 0$. The energy decreases or stays the same. Since there are finitely many states, the network must converge to a local minimum. $\square$

The visualization shows the energy trace during recall — watch it monotonically decrease until convergence.

---

Why It Works: Energy Minimization

The stored patterns sit at (or near) local minima of the energy function. To see why, let's compute the energy when the network is in the state of pattern $\nu$, i.e., $\mathbf{s} = \boldsymbol{\xi}^\nu$:

$$E(\boldsymbol{\xi}^\nu) = -\sum_{i<j} W_{ij} , \xi_i^\nu , \xi_j^\nu$$

Substituting the Hebbian weights:

$$= -\sum_{i<j} \frac{1}{N} \sum_{\mu=1}^{P} \xi_i^\mu \xi_j^\mu \cdot \xi_i^\nu \xi_j^\nu$$

$$= -\frac{1}{N} \sum_{\mu=1}^{P} \sum_{i<j} (\xi_i^\mu \xi_i^\nu)(\xi_j^\mu \xi_j^\nu)$$

For $\mu = \nu$ (the pattern itself), $\xi_i^\mu \xi_i^\nu = 1$ for all $i$, and the inner sum gives $\binom{N}{2} = \frac{N(N-1)}{2}$, contributing $-\frac{N-1}{2}$ to the energy.

For $\mu \neq \nu$ (other patterns), if patterns are random and uncorrelated, the products $\xi_i^\mu \xi_i^\nu$ are random $\pm 1$, and the sum is $O(\sqrt{N})$ — negligible compared to $N$.

So:

$$E(\boldsymbol{\xi}^\nu) \approx -\frac{N-1}{2} + O\left(\frac{P\sqrt{N}}{N}\right)$$

The first term is large and negative — the stored pattern has very low energy. Nearby states (a few neurons flipped) have slightly higher energy. The pattern sits at the bottom of an energy basin, and the update rule rolls downhill into it.

The energy landscape in the visualization shows exactly this. The x and y axes represent overlap with two stored patterns. Dark blue regions are low energy (basins). The cyan dot shows the current state rolling downhill during recall.

The picture to have in mind:

Energy
  ^
  |   ___         ___
  |  /   \       /   \
  | /     \_____/     \
  |/      P1    P2     \
  +-------------------------> State space

Each stored pattern $P_1, P_2$ creates a valley. Starting from any initial state, the network flows to the nearest valley.

---

Capacity: How Much Can It Remember?

A natural question: how many patterns can we store before the network starts making mistakes?

For random patterns of length $N$, the critical capacity is:

$$\boxed{P_{\max} \approx 0.138 , N}$$

This result, due to Amit, Gutfreund, and Sompolinsky (1985), means:

• Below $0.138N$ patterns: recall is nearly perfect
• Above $0.138N$ patterns: the basins of attraction shrink, overlap, and eventually disappear
• The network starts confusing patterns or converging to spurious mixtures

For our 400-neuron network: $P_{\max} \approx 55$ patterns.

The capacity is linear in $N$ — double the neurons, double the memory. But it's also quite small relative to the total number of possible patterns ($2^N$). The network uses its $\frac{N(N-1)}{2}$ weights (about 80,000 for $N=400$) to carve out just 55 stable valleys in a space of $10^{120}$ states.

The Overlap Parameter

To measure how well recall works, we use the overlap between the network state and a stored pattern:

$$m^\mu = \frac{1}{N} \sum_{i=1}^{N} s_i , \xi_i^\mu$$

This ranges from $-1$ (anti-correlated, inverted pattern) through $0$ (uncorrelated) to $+1$ (perfect match). Successful recall means $|m^\mu| \approx 1$.

The visualization shows overlap bars after convergence — 100% means the recalled pattern matches the stored pattern perfectly.

---

Failure Modes: Spurious States

Hopfield networks aren't perfect. The energy landscape has local minima that don't correspond to any stored pattern. These are called spurious states, and there are several kinds:

1. Inverted Patterns

If $\boldsymbol{\xi}^\mu$ is a stored pattern, then $-\boldsymbol{\xi}^\mu$ (every neuron flipped) is also a local minimum with the same energy. The network has no notion of "right side up."

2. Mixture States

With three stored patterns $\boldsymbol{\xi}^1, \boldsymbol{\xi}^2, \boldsymbol{\xi}^3$, there can be a spurious minimum near:

$$s_i = \text{sign}(\pm\xi_i^1 \pm \xi_i^2 \pm \xi_i^3)$$

These are "chimera" patterns — blends of stored memories. They appear when patterns share structure and their basins overlap.

3. Spin Glass States

At high load ($P$ close to $0.138N$), the energy landscape becomes rough and develops many shallow local minima unrelated to any stored pattern. The network behaves like a spin glass — a disordered magnetic system with many metastable states.

---

Connection to Physics

The Hopfield network is mathematically identical to an Ising model from statistical mechanics. Each neuron is a spin, the weights are coupling constants, and the energy function is the Hamiltonian:

$$H = -\sum_{i<j} J_{ij} \sigma_i \sigma_j$$

Neural Network	Statistical Physics
Neuron state $s_i$	Spin $\sigma_i$
Weight $W_{ij}$	Coupling $J_{ij}$
Energy $E$	Hamiltonian $H$
Recall	Relaxation to ground state
Stored pattern	Ferromagnetic ground state
Spurious state	Metastable state
Capacity transition	Phase transition

This connection is not just an analogy — it is an exact mathematical equivalence. The capacity result $P_{\max} = 0.138N$ was derived using the replica method from spin glass theory, the same technique used to study disordered magnets.

Hopfield's 1982 paper, along with the subsequent statistical mechanics analysis, helped establish the field of neural network theory and contributed to Hopfield receiving the 2024 Nobel Prize in Physics.

---

Workflow

1. Draw a pattern on the grid (or load a preset)
2. Store it — the weight matrix updates, a basin forms in the energy landscape
3. Repeat for a second pattern — watch the landscape develop two distinct valleys
4. Load one of the stored patterns
5. Add noise to corrupt it
6. Recall — watch the network converge back to the original pattern
7. Use Step mode to see individual neuron updates with the full inference breakdown
8. Select neurons to inspect their weights and contributions

Keyboard Shortcuts

Key	Action
Space	Start/stop recall
.	Single step
D	Draw mode
V	Select mode
C	Clear grid
N	Add noise
S	Store pattern
I	Invert pattern

---

References

• Hopfield, J. J. (1982). "Neural networks and physical systems with emergent collective computational abilities." Proceedings of the National Academy of Sciences, 79(8), 2554-2558.
• Amit, D. J., Gutfreund, H., & Sompolinsky, H. (1985). "Storing infinite numbers of patterns in a spin-glass model of neural networks." Physical Review Letters, 55(14), 1530.
• Hertz, J., Krogh, A., & Palmer, R. G. (1991). Introduction to the Theory of Neural Computation. Addison-Wesley.

---

Built with plain HTML, CSS, and JavaScript. No frameworks, no dependencies.