Scribd
Upload
17 views2 pages

Problem Set 8 (Due Thurs May 6) Models of Associative Memory

The document summarizes a problem set on models of associative memory in computational neuroscience. It includes 3 problems analyzing the capacity of different associative memory networks: 1) The original Hopfield model with {-1,1} units shows a capacity of around 0.14N for large N using MATLAB simulations. 2) A sparse network with {0,1} units and a covariance learning rule has a estimated capacity of P=N2f|log(f)|. 3) Another sparse {0,1} network with a different learning rule is also analyzed.

Uploaded by

DevendraReddyPoreddy
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
17 views2 pages

Problem Set 8 (Due Thurs May 6) Models of Associative Memory

The document summarizes a problem set on models of associative memory in computational neuroscience. It includes 3 problems analyzing the capacity of different associative memory networks: 1) The original Hopfield model with {-1,1} units shows a capacity of around 0.14N for large N using MATLAB simulations. 2) A sparse network with {0,1} units and a covariance learning rule has a estimated capacity of P=N2f|log(f)|. 3) Another sparse {0,1} network with a different learning rule is also analyzed.

Uploaded by

DevendraReddyPoreddy
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 2

MIT Department of Brain and Cognitive Sciences

9.29J, Spring 2004 - Introduction to Computational Neuroscience


Instructor: Professor Sebastian Seung

Problem Set 8 (due Thurs May 6)


Models of Associative Memory

April 29, 2004

1. Capacity of the Hopfield Model.


Consider the Hopfield model, using a sequential update, given by
�
si = sign( Wij sj )
j
�
1
� µ µ
N µ ξi ξj i �= j
Wij =
0 i=j

where s ∈ {−1, 1}N . (Sequential update means that the neurons are updated one at a time, typically in random
order.)
One definition of the capacity of the Hopfield model is the number of patterns that can be stored where some
small fraction (Perr ≤ 0.01) of the bits are corrupted. Using this definition, the capacity of the original Hopfield
model is approximately 0.14N for large N , where N is the number of units in the network. In this problem,
we will validate this capacity using a simple MATLAB simulation, and then use our simulation to compare the
capacities of original Hopfield model with the capacities of a network storing sparse patterns using {0, 1} units.

(a) Construct P random {−1, 1} patterns, ξ 1 , ..., ξ P , each of size N . Find W using the prescription given
above.

We investigate the capacity by checking if each of the stored patterns are actually steady states of the
system. The weight update from a stored pattern ξ ν can be written as:
1 �� µ µ ν
si = sign(ξiν + ξi ξj ξj ).
N
�
µ=ν �
j=i

We would like si to equal ξiν , but our steady state


� could be corrupted by the zero­mean crosstalk term. To
visualize this in MATLAB, collect the terms j Wij ξju for all i and all µ and make a histogram of the
results. To get a nice plot, use N = 1000 and 50 bins instead of MATLAB’s default of 10.

Submit your matlab code and plots for P = 100, 200, and 140 (the known capacity for N → ∞). Describe
in words how the shape of the histogram changes as we change P , and how this impacts the capacity.

2. Storing binary patterns using the covariance rule.


In this problem, we will consider a sparse network. This means that instead of the {−1, 1} network used in the
previous problem, we will use {0, 1} units. The patterns that we wish to store are random with each bit having
probability f of being a 1. We are interested in the case where f is small.
The network is defined by the covariance rule:
�
µ
1
− f )(ξjµ − f ) i �= j
�
N f (1−f ) µ (ξi
Wij =
0 i=j

1
with the discrete dynamics: �
xi = H( Wij xj − θi )
j

where H is the Heaviside function: H(u) = 1 if u > 0, otherwise H(u) = 0.

(a) Show that for large N and small f the sum j Wij ξjν can be separated into ξiν and a crosstalk term.
�

(b) Show that this crosstalk term has zero mean.


(c) Construct P random {0, 1} patterns, each of size N , using f as the probability of a 1 bit. Plot the histogram
of j Wij ξjµ as in part a. Experiment with P to estimate the capacity for N = 1000 and f = 0.05.
�

(d) According to your simulations, what value of the threshold θi maximizes the capacity?

(e) One published result estimates the capacity of the sparse network as P = 2f | log(f )| . How well does this
quantity compare to your results (test this by varying N and f )?

3. Storing binary patterns using another rule.


As in the previous problem, we will consider a network with {0, 1} units but with a different rule:
� � µ µ
1
(ξ ξ − f 2 ) i =� j
Wij = N f µ i j
0 i=j

with the discrete dynamics: �


xi = H( Wij xj − θi ).
j

(a) Repeat (a)­(d) from the previous problem using this network.
(b) Extra credit: Derive an expression for the capacity P in terms of N and f.

4. Compare the capacities of the 3 networks considered.

You might also like

pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy