BACKpropagation through BACK substitution with a BACKslash. (arXiv:2303.15449v1 [math.NA])

We present a linear algebra formulation of backpropagation which allows the
calculation of gradients by using a generically written “backslash” or
Gaussian elimination on triangular systems of equations. Generally the matrix
elements are operators. This paper has three contributions:

1. It is of intellectual value to replace traditional treatments of automatic
differentiation with a (left acting) operator theoretic, graph-based approach.

2. Operators can be readily placed in matrices in software in programming
languages such as Ju lia as an implementation option.

3. We introduce a novel notation, “transpose dot” operator
“${}^{T_bullet}$” that allows the reversal of operators.

We demonstrate the elegance of the operators approach in a suitable
programming language consisting of generic linear algebra operators such as
Julia cite{bezanson2017julia}, and that it is possible to realize this
abstraction in code. Our implementation shows how generic linear algebra can
allow operators as elements of matrices, and without rewriting any code, the
software carries through to completion giving the correct answer.