“AI is transforming the entire world of technology. Much of this progress is due to the ability of learning algorithms to spot patterns in larger and larger amounts of data. Today this is powering everything from web search to self-driving cars. This insatiable hunger for processing data has caused the bleeding edge of machine learning to shift from CPU computing, to cloud, to GPU, to HPC,” observes Andrew Ng, the Chief Scientist at Baidu.
Ng will describe this in more detail at his much-anticipated upcoming talk, How HPC in Supercharging Machine Learning at the June ISC High Performance conference in Frankfurt, Germany. To look ahead, we spoke to him to find out more about machine learning and how HPC fits into the machine learning ecosystem. Ng’s answers are offset.
TNP: Artificial intelligence has had a few false starts. What is different about machine learning?
Two of the key drivers of machine learning progress today are 1. scale of data and 2. scale of computation. With society spending more time on websites and mobile devices, data has been rapidly accumulating for the past two decades. It was only recently that researchers have figured out how to scale computation to build deep learning algorithms that can take effective advantage of this voluminous amount of data.
TNP: Which industries or application domains do you think will be first to feel the effects of machine learning in a significant way?
Machine learning has already transformed the internet with services such as web search, anti-spam and product recommendations. This is just the beginning. One of the areas I’m most excited about right now is speech recognition. Speech is one of those technologies with the potential to change everything. Today, speech recognition isn’t reliable enough for you to use all of the time but I hope in the near future we can create a new way of interacting with all your devices. The best technology is often invisible, and as speech recognition becomes more reliable, I hope it will disappear into the background.
I am also excited about autonomous driving. This is an area where technology and regulations should be developed simultaneously. I hope to see government organizations and tech communities working together on this. Making autonomous cars a reality cannot be done by any single organization. It will require a public-private partnership.
Sieslack: How does high performance computing fit into machine learning? What HPC technologies are most important to this ecosystem?
Although the techniques behind machine learning and deep learning have been studied for decades, they rely on large data sets and large computational resources, and so have only recently become practical for many problems. Training deep neural networks is very computationally intensive: training one of our models takes tens of exaflops of work, and so HPC techniques are key to creating these models.
Because the neural network training problem is so arithmetically intense, we rely on computationally dense processors like GPUs, and because we need to scale the training process over multiple nodes, we rely on fast interconnect technologies such as Infiniband. Along with HPC hardware, we also use HPC software such as MPI and BLAS libraries. Perhaps most importantly, we approach problems from an HPC point of view: we examine the fundamental limits to our computation, and then push to see how close we can get to those limits.
The faster we train our networks, the more iteration we can make on our datasets and models, and the more iterations we make, the more we advance our machine learning. This means that HPC translates into machine learning progress, which is why we have adopted the HPC point of view. Machine learning should embrace HPC. These methods will make researchers more efficient and help accelerate the progress of our whole field. At Baidu, Dr. Bryan Catanzaro leads a team that has been at the forefront of this field. I hope these ideas will disseminate into more AI teams around the world, and also that more HPC researchers will come help the field of AI.
Sieslack: What are the most interesting developments in the field of machine learning today?
AI has made tremendous progress, and I’m optimistic about building a better society that is embedded up and down with machine intelligence. But AI today is still very limited. Almost all the economic and social value of deep learning is through “supervised learning,” which is limited by the amount of suitably formatted (i.e., labeled) data. Looking ahead, there are many other types of AI beyond supervised learning that I find exciting, such as unsupervised learning (where we have a lot more data available, because the data does not need to be labeled). There’s a lot of excitement about these other forms of learning in our group and others.