Kanerva++: extending The Kanerva Machine with differentiable, locally block allocated latent memory. (arXiv:2103.03905v1 [cs.NE])

Episodic and semantic memory are critical components of the human memory
model. The theory of complementary learning systems (McClelland et al., 1995)
suggests that the compressed representation produced by a serial event
(episodic memory) is later restructured to build a more generalized form of
reusable knowledge (semantic memory). In this work we develop a new principled
Bayesian memory allocation scheme that bridges the gap between episodic and
semantic memory via a hierarchical latent variable model. We take inspiration
from traditional heap allocation and extend the idea of locally contiguous
memory to the Kanerva Machine, enabling a novel differentiable block allocated
latent memory. In contrast to the Kanerva Machine, we simplify the process of
memory writing by treating it as a fully feed forward deterministic process,
relying on the stochasticity of the read key distribution to disperse
information within the memory. We demonstrate that this allocation scheme
improves performance in conditional image generation, resulting in new
state-of-the-art likelihood values on binarized MNIST (<=41.58 nats/image) ,
binarized Omniglot (<=66.24 nats/image), as well as presenting competitive
performance on CIFAR10, DMLab Mazes, Celeb-A and ImageNet32x32.