System Software, Orchestration Gets an OpenHPC Boost
June 29, 2016 John Kirkley
System software setup and maintenance has become a major efficiency drag on HPC labs and OEMs alike, but community and industry efforts are now underway to reduce the huge amounts of duplicated development, validation and maintenance work across the HPC ecosystem. Disparate efforts and approaches, while necessary on some levels, slow adoption of hardware innovation and progress toward exascale performance. They also complicate adoption of complex workloads like big data and machine learning.
With the creation of the OpenHPC Community, a Linux Foundation collaborative project, the push is on to minimize duplicated efforts in the HPC software stack wherever possible and create a stable, flexible platform for fast-evolving HPC workloads. And with the release of Intel HPC Orchestrator software, Intel is aiming to further reduce the burdens of HPC setup and maintenance on labs and OEMs, by providing support across the system software stack for the HPC ecosystem.
Intel has been involved with OpenHPC from the beginning, seeding the software stack and committing to working closely with vendors, universities and national labs to evolve the system software stack and chart a more efficient path forward. In the process, Intel has also recognized that many HPC sites could benefit from added support for their stacks, hence the creation of the Intel HPC Orchestrator Product Family. While the OpenHPC software stack can provide a head start, if the HPC site wants the added support, has “custom” requirements or wants to update at a different speed, they are better off using Intel HPC Orchestrator.
Intel HPC Orchestrator is a family of modular Intel-licensed and supported premium products based on the publicly available OpenHPC software stack. Similar to the approach vendors such as SUSE and Red Hat take with Linux, Intel will harness community-innovation and provide additional upfront integration and validation for stack components, along with added troubleshooting tools and support.
As recently announced, the first available version, called Intel HPC Orchestrator – Advanced, is a flexible offering designed for Top 500 technical and commercial HPC users. “The goal with HPC Orchestrator – Advanced is to help our customers focus their time and energy on the things that are important and that they do best. So OEMs will be able to concentrate on customizing within or on top of the software stack to better differentiate their own products. ISVs and developers will be able to focus more on innovation and new efforts rather than constantly retesting apps for new system software releases. And system admins will be able to more easily manage clusters from multiple vendors while end users will have more time to focus on their scientific work and research,” said Figen Ulgen, General Manager, HPC Software and Cloud, Intel.
How Intel HPC Orchestrator builds on OpenHPC
According to Intel, the Intel HPC Orchestrator system software stack will deliver efficiency gains and greater peace of mind by providing additional software integration, testing, and validation on top of the base OpenHPC stack, along with proprietary Intel tools that can help simplify development efforts and troubleshooting. “With respect to integration and validation, we are performing advanced integration testing between components in Intel HPC Orchestrator, along with validation testing on a 1000-node system, including testing of specific platform configurations,” said Ulgen. “We are also performing rigorous scalability, reliability, performance and security testing on top of the standard tests that OpenHPC uses.”
Michael Miller, president of strategy, alliances and marketing at SUSE notes that these efforts are not only important to saving labs and end users time, they are key to faster overall progress toward exascale and other HPC innovations. “Together with Intel and the other founding members of OpenHPC, SUSE is accelerating the race toward making exascale computing a reality,” said Miller. “To get there faster, the industry needs to maximize every bit of hardware innovation available while focusing on faster innovation up the stack. This means we need stop wasting precious resources on duplication and multiple one-off versions of the infrastructure software stack. Intel HPC Orchestrator, powered by SUSE Linux* Enterprise Server for HPC, is built for exactly that purpose and provides a rigorously tested, highly scalable and secure software stack on which to build the next generation of HPC.”
Intel HPC Orchestrator includes a version of the complete Intel Parallel Studio XE Cluster Edition 2106 Suite (provided with 90-day evaluation license), which enables the development environment. Intel Cluster Checker, provided with a perpetual license, enhances software stack supportability, by creating system snapshots to show how a system has changed since the first time it was certified to help simplify troubleshooting.
According to initial announcements, Intel HPC Orchestrator will support the SUSE Linux Enterprise Server (SLES)12 SP1, Red Hat Enterprise Linux (RHEL) 7.2, and CentOS 7.2 operating systems at launch later this year. The system software stack includes image-based provisioning with Warewulf scripts; resource manager choices, including PBS Professional and Simple Linux Utility for Resource Management (Slurm); and multiple Message Passing Interface (MPI) families and development environments.
Support and services in the Intel HPC Orchestrator – Advanced offering include up to Level 2-3 support provided via the system OEM or integrator, for system software stack components and validated component updates. Intel is fully supporting its own products in the Intel HPC Orchestrator offerings, and will have support agreements with most of the commercially supported third-party providers of system software components to help ensure that all of the software and support is managed through one source, to provide convenience and fast service for system vendors and their end-user customers.
In addition to participating in OpenHPC, Intel has been investing heavily in making HPC more accessible and more efficient through the development of Intel Scalable System Framework (SSF). Intel HPC Orchestrator will provide optimized integration for Intel SSF, integrating the software dependencies across SSF components, including Intel Xeon and next-generation Intel Xeon Phi processors, Intel Omni-Path Fabric, and Intel Enterprise Edition for Lustre software, for example.
Is Intel really getting into the software support business?
If you’re wondering why Intel is making such a big investment in supporting software, it’s a good question. Ulgen explained: “Intel has the expertise and a vested interest in fostering faster adoption of hardware innovations in the HPC space, and we’re already heavily involved in many open source efforts. For example, Intel is one of the largest contributors to the Linux Kernel on a monthly basis. We’re using a similar model with Intel Solutions for Lustre Software where there is an open source community version and an Intel-supported product offering. With multiple engineering teams working on elements of the system software stack, we’ve drawn on the expertise of a team of more than 100 developers at times to move efforts forward. So our efforts around Intel HPC Orchestrator are really a natural extension of other things we’ve been doing and we will continue to invest in this direction.”
Intel has indicated that additional products in the Intel HPC Orchestrator family will be introduced down the road. The intent is to provide system software stack solutions that address the differing needs of the HPC industry, from large systems at the top of the Top500 supercomputer list to smaller, single workload optimized systems for enterprise use.
Intel’s efforts around Intel HPC Orchestrator are all about simplifying life for OEMs, systems integrators and system admins. “Users of the Intel HPC Orchestrator system software stack will be able to focus resources on differentiating capabilities or application innovations rather than duplicated efforts of maintaining the base system software,” said Ulgen “Between validated software updates and Level 3 technical support for software stack integration and key components, we’re hoping to give Intel HPC Orchestrator users a significant chunk of time back. And that should help the HPC ecosystem to move faster, as a whole, to exascale computing and innovation in areas such as machine learning, high-performance analytics and even HPC in the cloud.”