NEVIS’22: A Stream of 100 Tasks Sampled from 30 Years of Computer Vision Research. (arXiv:2211.11747v1 [cs.LG])

We introduce the Never Ending VIsual-classification Stream (NEVIS’22), a
benchmark consisting of a stream of over 100 visual classification tasks,
sorted chronologically and extracted from papers sampled uniformly from
computer vision proceedings spanning the last three decades. The resulting
stream reflects what the research community thought was meaningful at any point
in time. Despite being limited to classification, the resulting stream has a
rich diversity of tasks from OCR, to texture analysis, crowd counting, scene
recognition, and so forth. The diversity is also reflected in the wide range of
dataset sizes, spanning over four orders of magnitude. Overall, NEVIS’22 poses
an unprecedented challenge for current sequential learning approaches due to
the scale and diversity of tasks, yet with a low entry barrier as it is limited
to a single modality and each task is a classical supervised learning problem.
Moreover, we provide a reference implementation including strong baselines and
a simple evaluation protocol to compare methods in terms of their trade-off
between accuracy and compute. We hope that NEVIS’22 can be useful to
researchers working on continual learning, meta-learning, AutoML and more
generally sequential learning, and help these communities join forces towards
more robust and efficient models that efficiently adapt to a never ending
stream of data. Implementations have been made available at
https://github.com/deepmind/dm_nevis.