Python Snakes Along HPC Continuum

While Python has steadily been winding its way into an increasing number of enterprise settings over the last several years, in high performance computing, which tends to be dominated by lower-level programming approaches, it was overlooked in favor of a never-ending string of MPI and Fortran tweaks for performance reasons.

However, this has been changing—and with increased frequency in the last couple of years. This is, in part, due to the number of domain specialists in scientific computing who are coming into their fields with complex questions, an unprecedented slew of data, and the need to make tradeoffs between ultra-high performance on tens (or hundreds) of thousands of cores and simple productivity. The shift from more traditional languages in scientific computing is also being fed by the flood of new graduate students who are thinking about their algorithms with Python as the starting point. But ultimately, the ability to iterate fast and move forward quicker (even if the benchmarked performance with Fortran or C is better at the beginning, for instance) has its value in scientific computing circles, says Peter Wang, CTO and co-founder of Continuum Analytics, which produces its own supported Python distribution aimed specifically at the HPC set in research and commercial circles.

The Austin-based company was clearly onto something when it got its start in 2011. At the time, Python had already had its Cambrian explosion, but the blast didn’t hit the HPC set until a bit later. Continuum has raised $34 million overall, including $3 million in funds from DARPA to support the XDATA program. Specifically, the money was pushed their way to develop Python-based virtualization and interactive analysis of very large data sets.

The company had a strong story to tell for the XDATA initiative given its work on the NumPy package (Continuum’s CEO and co-founder, Travis Oliphant, was the original developer of that package) as well as SciPy, since these are both aimed at large-scale data analysis on the order of what national labs and universities require. Oliphant has not lost touch with his open source roots, either. He is currently the director of the Python Software Foundation and is a primary collaborator on the evolving toolset that makes Python a continued good fit for large-scale scientific data analysis. What Oliphant and Wang develop, in other words, are numerous Python-based tools, all of which must be brought under a single platform banner and optimized for a variety of HPC and large-scale data analysis environments, including Hadoop and accelerated systems.

For high performance computing, having all of this bundled with a rich set of operational, visualization, and other tools is important, but Python still has inroads to make to hit an even wider HPC base. Wang does not believe that Fortran and other lower level programming languages will fall by the wayside anytime soon, but he does believe that the rapid rise of Python elsewhere in the data analysis world will filter into HPC with increasing frequency. “HPC does not exist in a vacuum and Python is the fastest growing data analysis language in the world.”

In an effort to draw the diverse range of HPC users further into the Python fold, Wang’s company has already extended its reach into GPU computing with a complier for Python that hooks into Nvidia’s CUDA programming environment. Continuum worked closely with Nvidia for a number of years and helped unsure that Python was among the supported languages for CUDA. “There are other ways of getting to CUDA with Python,” Wang explains, “but they all involve learning CUDA. With the compiler framework we’ve developed, it boils down to adding a single line decorator on top of a function and we can auto-compile those functions to run on GPUs. This has been important in some key areas outside of scientific computing, in finance, in particular.”

Continuum has had a long-standing base among the Wall Street set with big banks like JPMorgan Chase and Bank of America using their supported distribution of Python, Anaconda. Further, Wang says that almost all of the national labs use Anaconda as well. After all, the goal of developing tooling and supported languages for scientific codes is to allow scientists to get back to their “day jobs” without worrying about becoming programmers. “For these users, especially at the labs, they are getting the performance they need, but more important, the ecosystem is there with all of the tooling around visualization, reproducibility, collaboration with notebooks, even when there are many thousands of cores.”

This week at the annual Supercomputing Conference, Continuum announced that similar support would be offered for AMD’s APUs as well, opening the scope of broader accelerated computing for scientific communities. Although it’s still hard to tell how much traction, if at all, AMD is having in HPC with its APUs, Wang says that they expect interest in the CPU-GPU hybrid processor to grow and their involvement with AMD to help open further access to Python in the process.

“AMD’s HSA architecture features shared memory, which means performance gains without having to understand the underlying architecture, as opposed to having CPU and GPU separate and having to know the tricks to divide and conquer those workloads. AMD also has a community of 60,000 developers. That, coupled with the fact that even though the CPUs on the APUs are farily underpowered for now, means that work we’ve done with Numba can start now and scale with increases in capability of those processors,” Wang says.

The goal of the Anaconda platform Continuum maintains is to make Python more extensible and usable for a broader HPC set but ultimately, Wang notes, the real story of Python in HPC is about productivity—not just about having more options for systems to get better performance, but creating an environment where users can iterate quickly and develop their ideas much faster over the long haul.

Sign up to our Newsletter

Featuring highlights, analysis, and stories from the week directly from us to your inbox with nothing in between.

Subscribe now

Be the first to comment

Leave a Reply

Your email address will not be published.


This site uses Akismet to reduce spam. Learn how your comment data is processed.