On Robust Numerical Solver for ODE via Self-Attention Mechanism. (arXiv:2302.10184v1 [cs.LG])

With the development of deep learning techniques, AI-enhanced numerical
solvers are expected to become a new paradigm for solving differential
equations due to their versatility and effectiveness in alleviating the
accuracy-speed trade-off in traditional numerical solvers. However, this
paradigm still inevitably requires a large amount of high-quality data, whose
acquisition is often very expensive in natural science and engineering
problems. Therefore, in this paper, we explore training efficient and robust
AI-enhanced numerical solvers with a small data size by mitigating intrinsic
noise disturbances. We first analyze the ability of the self-attention
mechanism to regulate noise in supervised learning and then propose a
simple-yet-effective numerical solver, AttSolver, which introduces an additive
self-attention mechanism to the numerical solution of differential equations
based on the dynamical system perspective of the residual neural network. Our
results on benchmarks, ranging from high-dimensional problems to chaotic
systems, demonstrate the effectiveness of AttSolver in generally improving the
performance of existing traditional numerical solvers without any elaborated
model crafting. Finally, we analyze the convergence, generalization, and
robustness of the proposed method experimentally and theoretically.