• FPGAs, OpenHMC Push SKA HPC Processing Capabilities

    June 13, 2017

    Astronomy is the oldest research arena, but the technologies required to process the massive amount of data created from radio telescope arrays represents some of the most bleeding-edge research in modern computer science.

    With an exabyte of data expected to stream off the Square Kilometer Array (SKA), teams from both the front and back ends of the project have major challenges ahead. One “small” part of that larger picture of seeing farther into the universe than ever before is moving the data from the various distributed telescopes into a single unified platform and data format. This means transferring data from …

    Read more

  • The Composable Systems Wave Is Rising

    June 8, 2017

    Hardware is, by its very nature, physical and therefore, unlike software or virtual hardware and software routines encoded by FPGAs, it is the one thing that cannot be easily changed. The dream of composable systems, which we have discussed in the past, is something that has been swirling around in the heads of system architects for more than a decade, and we are without question getting closer to realizing the dream of making the components of systems and the clusters that are created from them programmable like software.

    The hyperscalers, of course, have been on the bleeding edge of …

    Read more

  • Unifying Oil and Gas Data at Scale

    May 30, 2017

    The oil and gas industry has been among the most aggressive in pursuing internet of things (IoT), cloud and big data technologies to collect, store, sort and analyze massive amounts of data in both the drilling and refining sectors to improve efficiencies and decision-making capabilities. Systems are increasingly becoming automated, and sensors are placed throughout processes to send back data on various the systems and software has been put in place to crunch the data to create useful information.

    According to a group of researchers from Turkey, the oil and gas industry is well suited to embrace all the new …

    Read more

  • Singularity is the Hinge To Swing HPC Cloud Adoption

    May 18, 2017

    For almost a decade now, the cloud has been pitched as a cost-effective way to bring supercomputing out of the queue and into public IaaS or HPC on-demand environments. While there are certainly many use cases to prove that tightly-coupled problems can still work in the cloud despite latency hits (among other issues), application portability is one sticking point.

    For instance, let’s say you have developed a financial modeling application on an HPC on demand service to prove that the model works so you can make the case for purchasing a large cluster to run it at scale on-prem. This …

    Read more

  • Red Hat Gears Up OpenShift For Developers

    May 2, 2017

    During the five years that Red Hat has been building out its OpenShift cloud applications platform, much of the focus has been on making it easier to use by customers looking to adapt to an increasingly cloud-centric world for both new and legacy applications. Just as it did with the Linux operating system through Red Hat Enterprise Linux and related middleware and tools, the vendor has worked to make it easier for enterprises to embrace OpenShift.

    That has included a major reworking of the platform with the release of version 3.0 last year, which ditched Red Hat’s in-house technologies for …

    Read more

  • Cluster Management for Distributed Machine Learning at Scale

    May 1, 2017

    Over the last couple of decades, those looking for a cluster management platform faced no shortage of choices. However, large-scale clusters are being asked to operate in different ways, namely by chewing on large-scale deep learning workloads—and this requires a specialized approach to get high utilization, efficiency, and performance.

    Nearly all of the cluster management tools from the high performance computing community are being bent in the machine learning direction, but for production deep learning shops, there appears to be a DIY tendency. This is not as complicated as it might sound, given the range of container-based open source tools, …

    Read more

  • Docker Completes Its Platform With DIY Linux

    April 19, 2017

    It all started with a new twist on an old idea, that of a lightweight software container running inside Linux that would house applications and make them portable. And now Docker is coming full circle and completing its eponymous platform by opening up the tools to allow users to create their own minimalist Linux operating system that is containerized and modular above the kernel and that only gives applications precisely what they need to run.

    The new LinuxKit is not so much a variant of Linux as a means of creating them. The toolkit for making Linuxes, which was unveiled …

    Read more

  • Red Hat Tunes Up OpenShift For Legacy Code In Kubernetes

    April 14, 2017

    When Red Hat began building out its OpenShift cloud application platform more than five years ago, the open source software vendor found itself in a similar situation as others in the growing platform-as-a-service (PaaS) space: they were all using technologies developed in-house because there were no real standards in the industry that could be used to guide them.

    That changed about three years ago, when Google officials decided to open source the technology – called Borg – they were using internally to manage the search giant’s clusters and make it available to the wider community. Thus was born Kubernetes, …

    Read more

  • Singularity Containers for HPC, Reproducibility, and Mobility

    April 10, 2017 Rob Farber

    Containers are an extremely mobile, safe and reproducible computing infrastructure that is now ready for production HPC computing. In particular, the freely available Singularity container framework has been designed specifically for HPC computing. The barrier to entry is low and the software is free.

    At the recent Intel HPC Developer Conference, Gregory Kurtzer (Singularity project lead and LBNL staff member) and Krishna Muriki (Computer Systems Engineer at LBNL) provided a beginning and advanced tutorial on Singularity. One of Kurtzer’s key takeaways: “setting up workflows in under a day is commonplace with Singularity”.

    Many people have heard about code modernization and …

    Read more

  • Increasing HPC Utilization with Meta-Queues

    March 20, 2017 Ben Cotton

    Solving problems by the addition of abstractions is a tried and true approach in technology. The management of high-performance computing workflows is no exception.

    The Pegasus workflow engine and HTCondor’s DAGman are used to manage workflow dependencies. GridWay and DRIVE route jobs to different resources based on suitability or available capacity. Both of these approaches are important, but they share a key potential drawback: jobs are still treated as distinct units of computation to be scheduled individually by the scheduler.

    As we have written previously, the aims of HPC resource administrators and HPC resource users are sometimes at odds. …

    Read more

Previous Articles

Mellanox

The Next Platform Weekly

Tap the stack to painlessly subscribe for a weekly email from The Next Platform, featuring highlights, analysis, and stories from the week directly from us to your inbox with nothing in between.