A Gradient Estimator for Time-Varying Electrical Networks with Non-Linear Dissipation. (arXiv:2103.05636v1 [cs.LG])

We propose a method for extending the technique of equilibrium propagation
for estimating gradients in fixed-point neural networks to the more general
setting of directed, time-varying neural networks by modeling them as
electrical circuits. We use electrical circuit theory to construct a Lagrangian
capable of describing deep, directed neural networks modeled using nonlinear
capacitors and inductors, linear resistors and sources, and a special class of
nonlinear dissipative elements called fractional memristors. We then derive an
estimator for the gradient of the physical parameters of the network, such as
synapse conductances, with respect to an arbitrary loss function. This
estimator is entirely local, in that it only depends on information locally
available to each synapse. We conclude by suggesting methods for extending
these results to networks of biologically plausible neurons, e.g.
Hodgkin-Huxley neurons.

Source: https://arxiv.org/abs/2103.05636


Related post