arXiv Analytics

Sign in

arXiv:1703.00133 [cs.SE]AbstractReferencesReviewsResources

Easy over Hard: A Case Study on Deep Learning

Wei Fu, Tim Menzies

Published 2017-03-01Version 1

While deep learning is an exciting new technique, the benefits of this method need to be assessed w.r.t. its computational cost. This is particularly important for deep learning since these learners need hours (to weeks) to train the model. Such long CPU times limit the ability of (a) a researcher to test the stability of their conclusion via repeated runs with different random seeds; and (b)other researchers to repeat, improve, or even refute that original work. For example, recently, deep learning was used to find which questions in the Stack Overflow programmer discussion forum can be linked together. That system took 14 hours to execute. We show here that a very simple optimizer called DE (differential evolution) can achieve similar (and sometimes better). The DE approach terminated in 10 minutes; i.e. 84 times faster hours than deep learning. We offer these results as a cautionary tale to the software analytics community and suggest that not every new innovation should be applied without critical analysis. If researchers deploy some new and expensive process, that work should be baselined against some simpler and faster alternatives.

Comments: 12 pages, 6 figures, submitted to FSE2017
Categories: cs.SE, cs.LG
Related articles: Most relevant | Search more
arXiv:1306.2502 [cs.SE] (Published 2013-06-11)
Case Study Based Software Engineering Project Development: State of Art
arXiv:1408.6228 [cs.SE] (Published 2014-08-26)
Estimation of the new agile XP process model for medium-scale projects using industrial case studies
arXiv:1904.04104 [cs.SE] (Published 2019-04-08)
Software Engineering in Civic Tech: A Case Study about Code for Ireland