An Introduction to Automatic Differentiation forMachine Learning. (arXiv:2110.06209v1 [cs.LG])

Machine learning and neural network models in particular have been improving
the state of the art performance on many artificial intelligence related tasks.
Neural network models are typically implemented using frameworks that perform
gradient based optimization methods to fit a model to a dataset. These
frameworks use a technique of calculating derivatives called automatic
differentiation (AD) which removes the burden of performing derivative
calculations from the model designer. In this report we describe AD, its
motivations, and different implementation approaches. We briefly describe
dataflow programming as it relates to AD. Lastly, we present example programs
that are implemented with Tensorflow and PyTorch, which are two commonly used
AD frameworks.



Related post