Baidu’s Chief Scientist on Intersection of Supercomputing, Machine Learning

“AI is transforming the entire world of technology. Much of this progress is due to the ability of learning algorithms to spot patterns in larger and larger amounts of data. Today this is powering everything from web search to self-driving cars. This insatiable hunger for processing data has caused the bleeding edge of machine learning to shift from CPU computing, to cloud, to GPU, to HPC,” observes Andrew Ng, the Chief Scientist at Baidu.

Ng will describe this in more detail at his much-anticipated upcoming talk, How HPC in Supercharging Machine Learning  at the June ISC High Performance conference in Frankfurt, Germany. To look ahead, we spoke to him to find out more about machine learning and how HPC fits into the machine learning ecosystem. Ng’s answers are offset.

TNP: Artificial intelligence has had a few false starts. What is different about machine learning?

 Two of the key drivers of machine learning progress today are 1. scale of data and 2. scale of computation. With society spending more time on websites and mobile devices, data has been rapidly accumulating for the past two decades. It was only recently that researchers have figured out how to scale computation to build deep learning algorithms that can take effective advantage of this voluminous amount of data.

TNP: Which industries or application domains do you think will be first to feel the effects of machine learning in a significant way?

Machine learning has already transformed the internet with services such as web search, anti-spam and product recommendations. This is just the beginning. One of the areas I’m most excited about right now is speech recognition. Speech is one of those technologies with the potential to change everything. Today, speech recognition isn’t reliable enough for you to use all of the time but I hope in the near future we can create a new way of interacting with all your devices. The best technology is often invisible, and as speech recognition becomes more reliable, I hope it will disappear into the background.

I am also excited about autonomous driving. This is an area where technology and regulations should be developed simultaneously. I hope to see government organizations and tech communities working together on this. Making autonomous cars a reality cannot be done by any single organization. It will require a public-private partnership.

Sieslack: How does high performance computing fit into machine learning? What HPC technologies are most important to this ecosystem?

Although the techniques behind machine learning and deep learning have been studied for decades, they rely on large data sets and large computational resources, and so have only recently become practical for many problems. Training deep neural networks is very computationally intensive: training one of our models takes tens of exaflops of work, and so HPC techniques are key to creating these models.

Because the neural network training problem is so arithmetically intense, we rely on computationally dense processors like GPUs, and because we need to scale the training process over multiple nodes, we rely on fast interconnect technologies such as Infiniband. Along with HPC hardware, we also use HPC software such as MPI and BLAS libraries. Perhaps most importantly, we approach problems from an HPC point of view: we examine the fundamental limits to our computation, and then push to see how close we can get to those limits.

The faster we train our networks, the more iteration we can make on our datasets and models, and the more iterations we make, the more we advance our machine learning. This means that HPC translates into machine learning progress, which is why we have adopted the HPC point of view. Machine learning should embrace HPC. These methods will make researchers more efficient and help accelerate the progress of our whole field. At Baidu, Dr. Bryan Catanzaro leads a team that has been at the forefront of this field. I hope these ideas will disseminate into more AI teams around the world, and also that more HPC researchers will come help the field of AI.

Sieslack: What are the most interesting developments in the field of machine learning today?

AI has made tremendous progress, and I’m optimistic about building a better society that is embedded up and down with machine intelligence. But AI today is still very limited. Almost all the economic and social value of deep learning is through “supervised learning,” which is limited by the amount of suitably formatted (i.e., labeled) data. Looking ahead, there are many other types of AI beyond supervised learning that I find exciting, such as unsupervised learning (where we have a lot more data available, because the data does not need to be labeled). There’s a lot of excitement about these other forms of learning in our group and others.

Sign up to our Newsletter

Featuring highlights, analysis, and stories from the week directly from us to your inbox with nothing in between.
Subscribe now


  1. Very good article and certainly informative. The future of computing is bright and this article tells us that the application requirements continue to evolve which leads to HPC and computing industry needing to come up with better/creative/evolving ideas to improve performance. The nice thing about all of this is that one feeds the other and the market is only going to grow in the years ahead.

  2. As a tech guy, I appreciate the interest for autonomous driving. But my impression is that too many people take driving as a goal for itself and a fundamental need.

    What I notice personally, is that the better the Internet, the less I have to drive.

    I go to the office, only to talk to most of my colleagues and my boss via Lync/Skype/whatnot, so I wonder how long until they notice that they can watch me just as well in my home office.

    Can’t wait for a VR chats with my colleagues, showing them the nice new body I designed rather than the one I tried to salvage driving to the gym.

    BTW: Driving a 400hp car via my Oculus in Dirt Rally the other night, proved that I’m not fit for a major upgrade in horse power, so why bother putting in extra hours to pay for one?

    Lots of the spending side is also over the Internet, with delivery going to robots. Can’t say I’ll miss family shopping.

    Autonomous cars sounds like a solution to a problem fast disappearing or simply short sighted.

Leave a Reply

Your email address will not be published.


This site uses Akismet to reduce spam. Learn how your comment data is processed.