When SIMPLE is better than complex: A case study on deep learning for predicting Bugzilla issue close time. (arXiv:2101.06319v1 [cs.SE])

Is deep learning over-hyped? Where are the case studies that compare
state-of-the-art deep learners with simpler options? In response to this gap in
the literature, this paper offers one case study on using deep learning to
predict issue close time in Bugzilla.

We report here that a SIMPLE extension to a decades-old feedforward neural
network works better than the more recent, and more elaborate, “long-short term
memory” deep learning (which are currently popular in the SE literature).
SIMPLE is a combination of a fast feedforward network and a hyper-parameter
optimizer. SIMPLE runs in 3 seconds while the newer algorithms take 6 hours to
terminate. Since it runs so fast, it is more amenable to being tuned by our
optimizer. This paper reports results seen after running SIMPLE on issue close
time data from 45,364 issues raised in Chromium, Eclipse, and Firefox projects
from January 2010 to March 2016. In our experiments, this SIMPLEr tuning
approach achieves significantly better predictors for issue close time than the
more complex deep learner. These better and SIMPLEr results can be generated
2,700 times faster than if using a state-of-the-art deep learner.

From this result, we make two conclusions. Firstly, for predicting issue
close time, we would recommend SIMPLE over complex deep learners. Secondly,
before analysts try very sophisticated (but very slow) algorithms, they might
achieve better results, much sooner, by applying hyper-parameter optimization
to simple (but very fast) algorithms.

Source: https://arxiv.org/abs/2101.06319


Related post