Deep Learning Framework Power ScoresĀ 2018

By Jeff Hale

Deep learning continues to be the hottest thing in data science. Deep learning frameworks are changing rapidly. Just five years ago, none of the leaders other than Theano were even around.

I wanted to find evidence for which frameworks merit attention, so I developed this power ranking. I used 11 data sources across 7 distinct categories to gauge framework usage, interest, and popularity. Then I weighted and combined the data in this Kaggle Kernel.

UPDATE SEPT 20, 2018: By popular demand I expanded the frameworks evaluated to include Deeplearning4J. This changed the scores minimally.

Without further ado, here are the Deep Learning Framework Power Scores:

While TensorFlow is the clear winner, there were some surprising findings. Let’s dive in!

All of these frameworks are open source. All except one work with Python, and some can work with R or other languages.

TensorFlow is the undisputed heavyweight champion. It has the most GitHub activity, Google searches, Medium articles, books on Amazon and ArXiv articles. It also has the most developers using it and is listed in the most online job descriptions. TensorFlow is backed by Google.

Keras has an “API designed for human beings, not machines.” It is the second most popular framework in nearly all evaluation areas. Keras sits on top of TensorFlow, Theano, or NLTK. Start with Keras if you are new to deep learning.

PyTorch is the third most popular overall framework and the second most popular stand-alone framework. It is younger than TensorFlow and has grown rapidly in popularity. It allows customization that TensorFlow does not. It has the backing of Facebook.

Theano was developed at the University of Montreal in 2007 and is the oldest significant Python deep learning framework. It has lost much of its popularity and it’s leader stated that major releases were no longer on the roadmap. However, updates continue to be made. Theano still shows up in job listings somewhat frequently.

MXNET is incubated by Apache. It is close to eclipsing Theano to become the fourth most popular deep learning library. It has a fair number of contributors, search results, books, and scholarly articles. But its reported usage is down from last year and there aren’t a lot of folks searching for it.

CNTK is the Microsoft Cognitive Toolkit. It reminds me of many other Microsoft products in the sense that it is trying to compete with Google and Facebook and is not winning significant adoption.

Deeplearning4J, also called DL4J, is used with the Java language. It’s the only semi-popular framework not available in Python. However, you can import models written with Keras to DL4J.

I added DL4J to the analysis just after this article was originally published, so it received four extra days for data to accumulate. This was the only framework where two different search terms occasionally had different results. I used the higher number for each metric. As the framework scored quite low, this made no material difference.

FastAI is built on PyTorch. Its api was inspired by Keras and requires even less code for strong results. FastAI is bleeding edge as of mid-Sept 2018. It’s undergoing a rewrite for version 1.0 slated for October 2018 release. Jeremy Howard, the force behind FastAI has been a top Kaggler and President of Kaggle. He discusses why FastAI switched from Keras to make their own framework here.

You could ask why FastAI is on this list at all. It is not yet in demand for careers nor is it being used widely. However, it has a large built-in pipeline of users through its popular free online courses. It is also both powerful and easy to use. Its adoption could grow significantly.

Caffe was excluded from this list because it was evolved into Caffe2 and then Caffe2 was merged into PyTorch in March 2018.

I chose the following categories to provide a well-rounded view of popularity and interest in deep learning frameworks.

The evaluation categories are:

  • Online Job Listings
  • KDnuggets Usage Survey
  • Google Search Volume
  • Medium Articles
  • Amazon Books
  • ArXiv Articles
  • GitHub Activity

Searches were performed Sept. 16 to Sept. 20, 2018. Source data is in this Google sheet.

I used the plotly data visualization library and Python’s pandas library to explore popularity. For the interactive plotly charts, see my Kaggle Kernel here.

What deep learning libraries are in demand in today’s job market? I searched job listings on LinkedIn, Indeed, Simply Hired, Monster, and Angel List.

TensorFlow is the clear winner when it comes to frameworks mentioned in job listings. Learn it if you want a job doing deep learning.

I searched using the term machine learning followed by the library name. So TensorFlow was evaluated with machine learning TensorFlow. I tested several search methods and this one gave the most relevant results.

An additional keyword was necessary to differentiate that framework from unrelated terms because I was originally planning to include the Caffe library in this analysis.

KDnuggets, a popular data science website, polled data scientists around the world on the software that they used. They asked:

What Analytics, Big Data, Data Science, Machine Learning software you used in the past 12 months for a real project?

Here are the results for the frameworks in this category.

Keras showed a surprising amount of use — nearly as much as TensorFlow. It’s interesting that US employers are overwhelmingly looking for TensorFlow skills, when — at least internationally — Keras is used almost as frequently.

This category is the only one that includes international data because it would have been cumbersome to include international data for the other categories.

KDnuggets reported several years of data. While I used 2018 data only in this analysis, I should note that Theano, MXNET, and CNTK saw usage fall since 2017.

Web searches on the largest search engine are a good gauge of popularity. I looked at search history in Google Trends over the past year. Google doesn’t provide absolute search numbers, but it does provide relative figures.

TensorFlow held steady at 78 out of 100 as of the week ended Sept. 15, 2018. PyTorch is approaching Keras. Meanwhile the other frameworks have relative search volume scores at or below three. These scores were used for the power scores for this category.

Let’s look briefly at how search volume has changed over time to provide more historical context. The chart from Google directly below shows searches over the past two years.

TensorFlow = blue, Keras = red, PyTorch = yellow, Theano = Green

TensorFlow, Keras, and PyTorch searches all grew rapidly in late 2016 and 2017. Interestingly, none have been growing rapidly in the past several months. We may be seeing some seasonality, with drops in summer and winter holidays.

Google Trends allows only five terms to be compared simultaneously, so the other libraries were compared on a separate chart. None of the other libraries showed any thing other than minimal search interest relative to TensorFlow.

All these results are for searches in the US. Although the search terms Theano and Keras have other meanings, they are rarely used in the US outside ancient Greek classes.

I included several publication types in the power score. Older libraries have an advantage with more time to accumulate publications that feature them. Future power scores will focus on changes in publication numbers.

Let’s look at Medium articles first.

Medium is the place for popular data science articles and guides. And you’re here now — fantastic!

Finally a new winner. In terms of mentions in Medium articles, Keras broke the tape ahead of TensorFlow, but it was almost a photo finish. FastAI outperformed relative to its usual showing.

I hypothesize that these results might have occurred because Keras and FastAI are beginner friendly. They have quite a bit of interest from new deep learning practitioners, and Medium is often a forum for tutorials.

Now let’s look at which frameworks have books about them available on Amazon.

I searched for each deep learning framework on under Books->Computers & Technology.

TensorFlow for the win again. MXNET had more books than expected and Theano had fewer. PyTorch had relatively few books, but that may be because of the framework’s youth. This measure is biased in favor of older libraries because of the time it takes to publish a book.

Maybe Theano will fare better with more scholarly articles.

ArXiv is the online repository where most scholarly machine learning articles are published. I searched for each framework on the ArXiv website.

More of the same from TensorFlow for scholarly articles. The biggest surprise here might be that there aren’t more articles with these frameworks mentioned in total. Notice how much more popular Keras was on Medium and Amazon than in scholarly articles. Theano didn’t really perform well in this area given it’s age.

Activity on GitHub is another indicator of framework popularity. I broke out stars, forks, watchers, and contributors because they make more sense separately than combined.

TensorFlow is clearly the most popular framework on GitHub, with a whole lot of engaged users. FastAI has a decent following considering it isn’t even a year old. It’s interesting to see that contributor levels are closer for all of the frameworks than the other three metrics.

After gathering and analyzing the data it was time to consolidate it into one metric.

Here’s how I created the power score:

  1. Scaled all features between 0 and 1.
  2. Aggregated Job Search Listings and GitHub Activity subcategories.
  3. Weighted categories according to the weights below.

As shown above, Online Job Listings and KDnuggets Usage Survey make up half of the total score, while web searches, publications, and GitHub attention make up the other half. This split seemed like the most appropriate balance of the various categories.

4. Multiplied weighted scores by 100 for comprehensibility.

5. Summed category scores for each framework into a single power score.

Here’s the raw data:

Here are the scores after weighting and aggregating the subcategories.

And here’s the pretty chart again showing the final power scores.

100 is the highest possible score, indicating first place in every category. TensorFlow nearly achieved a 100, which was not surprising after seeing it at or near the top of every category.

To play with the charts interactively or fork the Jupyter Notebook, please head to this Kaggle Kernel.

For now, TensorFlow is firmly on top. It seems likely to continue to dominate in the short term. Given how quickly things move in the deep learning world though, that may change.

Time will tell if PyTorch surpasses TensorFlow as React surpassed Angular. The frameworks may be analogous. Both PyTorch and React are flexible frameworks backed by Facebook and often considered easier to use than their Google-backed competitors.

Will FastAI gain users outside its courses? It has a large pipeline of students, an even easier to use API than Keras. If you use FastAI, please let me know in the comments.

What do you think the future holds? Please share your thoughts below.

If you are considering learning one of these frameworks and have Python, numpy, pandas, sklearn, and matplotlib skills, I suggest you start with Keras. It has a large user base, is in demand by employers, has lots of articles on Medium, and has an API that is easy to use.

If you already know Keras, it might be tricky to decide on the next framework to learn. I suggest you pick either TensorFlow or PyTorch and learn it well so you can make great deep learning models.

TensorFlow is clearly the framework to learn if you want to master what is in demand. But PyTorch’s ease of use and flexibility make it friendlier to use.

Once you have those frameworks under your belt, I suggest you keep an eye on FastAI. Check out its free online course if you want to learn both basic and advanced deep learning skills. FastAI 1.0 promises to allow you to implement the latest deep learning strategies easily and to iterate quickly.

No matter which framework you choose, I hope you now have a better understanding of which deep learning frameworks are most in demand, most in use, and most written about.