A Survey of Deep Learning Frameworks
Currently a limited variety of tools are available in terms of deep learning frameworks since they implement algorithms which are used in bleeding edge applications such as computer vision and machine translation. It is necessary to select the proper framework for proper modelling of deep neural networks. This paper discusses the overview of deep learning and open source frameworks such as TensorFlow, CNTK, Theano, Torch, Caffe, MXnet and Neon.
1. Introduction:
Deep learning at its core can be considered to be a sub class of machine learning that has come to prominence in the last few years with the growth of cores available on GPUs for computing. Machine Learning in turn can be defined as a sub field of Artificial Intelligence which is concerned with developing algorithms which can aid to make data driven predictions or decisions.
In the field of deep learning currently there are several tools used by the industry to power applications such as self-driving cars to automatically generating captions for videos and images. As deep learning is the bleeding edge of current machine learning research most of the deep learning frameworks are open source and there appears to be little to no proprietary frameworks as of today.
Section 1 gives a brief introduction followed by section 2 which gives an overview of deep learning. Section 3 is a primer about the current open source deep learning frameworks. Finally, Section 4 provides a summary about the popular deep learning frameworks.
2. Overview of Deep Learning:
Deep learning algorithms are constituted of Artificial Neural Networks with hidden layers, which in turn were inspired by biological neural networks. [data01] Deep neural networks can be used to model complex non-linear relationships in both supervised (where historical data is used to make prediction about future outcomes) and unsupervised (clustering and finding new patterns and anomalies in data) settings. [gop01] Compared to traditional machine learning algorithms, deep learning models can provide significant improvement in areas such speech recognition and language translation as evidenced by the significant improvement in Google Translate after switching from Phrase Based Machine Translation (PBMT) to Neural Machine Translation (NMT). [obs01] The different type of Deep Learning models includes Deep Autoencoders, Restricted Boltzmann Machines (RBMs), Convolutional Neural Networks (CNNs), Recurrent Neural Networks and Long Short-Term Memory (LSTM) models depending on whether the application domain is for supervised or unsupervised learning. [Deng&Yu]
The next section describes the popular frameworks among data scientists and data engineers for production environments today. It should be noted that this list may not be exhaustive since listing of all the frameworks available would be difficult given the time and space for this survey.
3. Deep Learning Frameworks:
Given below are some of the popular open source deep learning frameworks of today. They have been categorized on languages supported, availability of tutorials and training materials, convolutional neural network modeling capability, recurrent neural network modeling capability, ease of use in terms of architecture, speed and support for multiple GPUs.
3.1 Theano:
Developed at the LISA lab at the University of Montreal, Theano consists of a Python library which allows users to define, optimize, and evaluate mathematical expressions on arrays and tensors. [deep01] It also generates customized C code for various mathematical operations. In terms of architecture, when compared to other deep learning frameworks available the popular consensus is that Theano is difficult to use. [data01] Theano also has multiple GPU support and provides support for convolutional neural networks and recurrent neural networks. Theano has been deployed in production in AI services companies such as ParallelDots serving up to a few 1000 concurrent news recommendations and multiple client Deep Learning as a service calls. [quor01]
3.2 TensorFlow:
TensorFlow was originally developed by researchers and engineers whom worked on the Google Brain Team within Google’s Machine Intelligence research organization for the purposes of conducting machine learning and deep neural networks research and over time the framework has been found to be general enough to be applicable in a wide variety of other domains as well. [Deng&Yu] TensorFlow is implemented in Python and uses data flow graphs for numerical computation. When compared to other deep learning frameworks it can be seen that TensorFlow is considered to currently be the best documented open source framework available. [data01] TensorFlow also boasts of an easy to use and modular front-end in terms of architecture. TensorFlow also provides supports for Convolutional Neural Networks and Recurrent Neural Networks. It should also be noted that there are implementations for Restricted Boltzmann Machines, Deep Autoencoders and Long Short-Term models utilizing TensorFlow architecture as well. [gith01] In addition, TensorFlow also contains TensorBoard which is a suite of visualization tools which make it easier to understand, debug and optimize programs which run on TensorFlow code.
3.3 Torch:
Torch is a deep learning framework with support for algorithms that give priority to GPUs. [torc01] Torch provides faster performance compared to other deep learning frameworks due to the use of the fast scripting language LuaJIT and its underlying C/CUDA implementation. Torch also possesses a large ecosystem of community driven packages and is also embeddable with iOS and Android backends. Torch also possesses Recurrent Neural Network and Convolution Neural Network modelling capability with a relatively easy to use modular front end. Torch is in use in companies such as Facebook, Google and Twitter as well as across research labs such as NYU, IDIAP and Purdue.
3.4 Caffe:
Caffe is a deep learning framework developed by Berkeley AI Research (BAIR) as well as community collaborators. It is implemented in C++. Caffe is released under the BSD 2- Clause license. [caff01] While Caffe supports Convolutional Neural Networks, it does not currently support Recurrent Neural Networks. Furthermore, even after extensive review of literature it was not possible to find any reference of many major players in the AI space deploying Caffe in a production environment. However Facebook recently released Caffe2 in April, which is a production ready cross-platform network and has been declared the successor to Caffe.
3.5 MXNet:
MXNet is a deep learning framework developed by collaborators from various companies and universities including the likes of Microsoft, Nvidia, Baidu, Intel, Carnegie Mellon University, University of Alberta and University of Washington. [mxne01] MXNet supports a plethora of programming languages including R, Python, Julia and Scala. It also has advanced GPU support compared to the other frameworks and also is relatively fast with regard to run time of deep learning algorithms. MXNet also has Convolutional Neural Network and Recurrent Neural Network modelling capabilities as well. MXNet is also Amazon Web Services (AWS)’s deep learning framework of choice.
3.6 Neon:
Neon is Intel Nervana’s reference deep learning framework which has been designed for extensibility and ease of use. [neon01] Neon supports Python and supports deep learning models such as Convolutional Neural Networks, Recurrent Neural Networks, Long Short-Term Memory (LSTM) Models and Deep Autoencoders. Furthermore, Neon is also tightly integrated with Intel’s GPU kernel library.
3.7 CNTK:
CNTK is the abbreviation for Microsoft Cognitive Toolkit which is an open source, commercial grade toolkit that can be used for training deep learning algorithms. [micr01] CNTK supports both C++ and Python. It also provides GPU support and is relatively fast compared to other deep learning frameworks. It has significant Recurrent Neural Network modelling capabilities as well as the ability to model Convolutional Neural Networks. Currently CNTK is used in production in Microsoft products such as Skype, Cortana, Bing and Xbox.
4. Summary:
Due to the variety of toolkits and frameworks available research is necessary to decide the most beneficial deep learning framework also taking into consideration the documentation available, and the supported languages. Most frameworks provide common deep learning algorithms suitable for a variety of use cases but may not have an easy to use and modular architecture and may also sacrifice speed. It is necessary to have sound knowledge of these frameworks in order that the necessary pros and cons can be evaluated for proper implementation in a production environment.
References:
[data01]
[1]
K. Waehner, “Open Source Deep Learning Frameworks and Visual Analytics,” [Online]. Available: http://www.datasciencecentral.com/profiles/blogs/open-source-deep-learning-frameworks-and-visual-analytics. [Accessed 15 July 2017].
[gop01]
[2]
J. O. “Models, Frameworks Accelerate Deep Learning for Enterprise Big Data,” [Online]. Available: https://goparallel.sourceforge.net/models-frameworks-accelerate-deep-learning-enterprise-big-data/. [Accessed 15 July 2017].
[obs01]
[3]
Observer, “Google Is Using Artificial Intelligence to Make a Huge Change to Its Translate Tool,” 06 03 2017. [Online]. Available: http://observer.com/2017/03/google-translate-neural-update/. [Accessed 28 97 2017].
[Deng&Yu]
[4]
L. Deng and D. Yu, Deep Learning: Methods and Applications, 2014.
[deep01]
[5]
“Theano at a Glance,” [Online]. Available: http://deeplearning.net/software/theano/. [Accessed 15 July 2017].
[quor01]
[6]
M. Mayank, “As of September 2015, is it better to use Torch or Theano for writing deep learning applications in production (not prototyping, not research, but production)?,” Quora, 15 09 2015. [Online]. Available: https://www.quora.com/As-of-September-2015-is-it-better-to-use-Torch-or-Theano-for-writing-deep-learning-applications-in-production-not-prototyping-not-research-but-production. [Accessed 28 07 2017].
[tens01]
[7]
Google Inc., “TensorFlow: An open-source software library for Machine Intelligence,” [Online]. Available: https://www.tensorflow.org/. [Accessed 7 July 2017].
[gith01]
[8]
G. Angeletti, “Deep Learning algorithms with TensorFlow,” Github, 16 04 2016. [Online]. Available: https://github.com/blackecho/Deep-Learning-TensorFlow. [Accessed 28 07 2017].
[torc01]
[9]
Torch, “Torch: A Scientific Computing Framework for LuaJIT,” [Online]. Available: http://torch.ch/. [Accessed 28 07 2017].
[caff01]
[10]
Y. Jia , “Caffe,” [Online]. Available: http://caffe.berkeleyvision.org/. [Accessed 29 07 2017].
[mxne01]
[11]
DMLC, “MXNet: A Flexible and Efficient Library for Deep Learning,” [Online]. Available: http://mxnet.io/. [Accessed 29 07 2017].
[neon01]
[12]
Nervana Systems Inc, “neon,” [Online]. Available: http://neon.nervanasys.com/docs/latest/. [Accessed 29 07 2017].
[micr01]
[13]
Microsoft, “The Microsoft Cognitive Toolkit,” [Online]. Available: https://www.microsoft.com/en-us/cognitive-toolkit/. [Accessed 29 07 2017].
