Learning Without a Global Clock: Asynchronous Learning in a Physics-Driven Learning Network. (arXiv:2201.04626v1 [cond-mat.soft])

In a neuron network, synapses update individually using local information,
allowing for entirely decentralized learning. In contrast, elements in an
artificial neural network (ANN) are typically updated simultaneously using a
central processor. Here we investigate the feasibility and effect of
asynchronous learning in a recently introduced decentralized, physics-driven
learning network. We show that desynchronizing the learning process does not
degrade performance for a variety of tasks in an idealized simulation. In
experiment, desynchronization actually improves performance by allowing the
system to better explore the discretized state space of solutions. We draw an
analogy between asynchronicity and mini-batching in stochastic gradient
descent, and show that they have similar effects on the learning process.
Desynchronizing the learning process establishes physics-driven learning
networks as truly fully distributed learning machines, promoting better
performance and scalability in deployment.

Source: https://arxiv.org/abs/2201.04626


Related post