MPI Archives - The Next Platform

Taking the Heavy Lifting Out of TensorFlow at Extreme Scale

April 25, 2017 Nicole Hemsoth

There is no real middle ground when it comes to TensorFlow use cases. Most implementations take place either in a single node or at the drastic Google-scale, with few scalability stories in between.

This is starting to change, however, as more users find an increasing array of open source tools based on MPI and other approaches to hop to multi-GPU scalability for training, but it still not simple to scale Google’s own framework across larger machines. Code modifications get hairy beyond single node and for the MPI uninitiated, there is a steep curve to scalable deep learning.

Although high performance …
Read more
Scaling Deep Learning on an 18,000 GPU Supercomputer

March 28, 2017 Nicole Hemsoth

It is one thing to scale a neural network on a single GPU or even a single system with four or eight GPUs. But it is another thing entirely to push it across thousands of nodes. Most centers doing deep learning have relatively small GPU clusters for training and certainly nothing on the order of the Titan supercomputer at Oak Ridge National Laboratory.

The emphasis on machine learning scalability has often been focused on node counts in the past for single-model runs. This is useful for some applications, but as neural networks become more integrated into existing workflows, including those …
Read more
Pushing MPI into the Deep Learning Training Stack

February 7, 2017 Nicole Hemsoth

We have written much about large-scale deep learning implementations over the last couple of years, but one question that is being posed with increasing frequency is how these workloads (training in particular) will scale to many nodes. While different companies, including Baidu and others, have managed to get their deep learning training clusters to scale across many GPU-laden nodes, for the non-hyperscale companies with their own development teams, this scalability is a sticking point.

The answer to deep learning framework scalability can be found in the world of supercomputing. For the many nodes required for large-scale jobs, the de facto …
Read more
MPI and Scalable Distributed Machine Learning

July 13, 2016 Rob Farber

MPI (Message Passing Interface) is the de facto standard distributed communications framework for scientific and commercial parallel distributed computing. The Intel MPI implementation is a core technology in the Intel Scalable System Framework that provides programmers a “drop-in” MPICH replacement library that can deliver the performance benefits of the Intel Omni-Path Architecture (Intel OPA ) communications fabric plus high core count Intel Xeon and Intel Xeon Phi processors.

“Drop-in” literally means that programmers can set an environmental variable to dynamically load the highly tuned and optimized Intel MPI library – no recompilation required! Of course, Intel’s MPI library supports other …
Read more
Mitigating MPI Message Matching Issues

June 27, 2016 Rob Farber

Since the 1990s, MPI (Message Passing Interface) has been the dominant communications protocol for high-performance scientific and commercial distributed computing. Designed in an era when processors with two or four cores were considered high-end parallel devices, the recent move to processors containing tens to a few hundred cores (as exemplified by the current Intel Xeon and Intel Xeon Phi processor families) has exacerbated scaling issues inside MPI itself. Increased network traffic, amplified by high performance communications fabrics such as InfiniBand and Intel Omni-Path Architecture (Intel OPA) manifest an MPI performance and scaling issue.

In recognition of their outstanding research and …
Read more
Intel Stretches Deep Learning on Scalable System Framework

May 10, 2016 Rob Farber

The strong interest in deep learning neural networks lies in the ability of neural networks to solve complex pattern recognition tasks – sometimes better than humans. Once trained, these machine learning solutions can run very quickly – even in real-time – and very efficiently on low-power mobile devices and in the datacenter.

However training a machine learning algorithm to accurately solve complex problems requires large amounts of data that greatly increases the computational workload. Scalable distributed parallel computing using a high-performance communications fabric is an essential part of what makes the training of deep learning on large complex datasets …
Read more
A Hybrid Path to Programming Exascale Machines

March 8, 2016 Nicole Hemsoth

As we have been describing here in detail, there is little end in sight to the train of exascale computing challenges ahead. The most prominent include building dense, energy-efficient systems that are capable of providing a 20X increase in floating point capability between now and the 2020-2024 timeframe, but more important, do so in just a tick above current power limits. For instance, today’s top supercomputer, the Tianhe-2 machine in China, consumes 18 megawatts while the goal for exascale systems is a peak power pull of 20 megawatts.

Massive power-efficient performance boosts aside, all of this effort will be useless …
Read more
Mellanox Turns InfiniBand Switches Into MPI Accelerators

November 12, 2015 Timothy Prickett Morgan

One of the key themes to improving the performance of clusters running simulations has been the offloading of common routines from the central processors in the servers to accelerators in the network adapter cards that plug into the servers and that interface with switches.

By moving such work from the server CPUs to the processing capability in the adapter cards, these functions can be radically accelerated while at the same time freeing up the CPUs to do other useful work, such as the number-crunching that is at the heart of the simulations. One of the most important offload functions that …
Read more
Evolving the HPC Software Stack with Fresh Hardware

November 11, 2015 Andrew Jones

There is a gentle touch of hypocrisy about the supercomputing world. We proudly, and rightly, advocate that the ability of HPC to deliver results to science, engineering and business and this relies not just on hardware but also on software and people. However, many of us still easily have our attention grabbed by the hardware – future processors, new architectures, esoteric options such as quantum computing – leaving the needs of programming, user interfaces, skills development, careers and diversity hiding at the back of the room.

When planning the development or enhancement of application software to use HPC facilities, or …
Read more
Exascale HPC Needs An Application Innovation Spark

September 29, 2015 Timothy Prickett Morgan

Every evolution in computing hardware brings with it big challenges for software developers.

Hyperscalers deal with scale, to be sure, but a lot of their work is just chunking through data in batch mode. At the upper echelons of the supercomputing market, the challenges are arguably the toughest software problems to tackle. HPC shops have to deal with scale and at the same time very complex models and simulations. The situation is not getting easier, but harder.

So the big question is this: What is going to happen when the supercomputer centers of the world have to port their codes …
Read more