• Increasing HPC Utilization with Meta-Queues

  March 20, 2017 Ben Cotton

  

  Solving problems by the addition of abstractions is a tried and true approach in technology. The management of high-performance computing workflows is no exception.

  The Pegasus workflow engine and HTCondor’s DAGman are used to manage workflow dependencies. GridWay and DRIVE route jobs to different resources based on suitability or available capacity. Both of these approaches are important, but they share a key potential drawback: jobs are still treated as distinct units of computation to be scheduled individually by the scheduler.

  As we have written previously, the aims of HPC resource administrators and HPC resource users are sometimes at odds. …

  Read more

• Docker Reaches The Enterprise Milestone

  March 2, 2017 Timothy Prickett Morgan

  

  In the server virtualization era, there were a couple of virtual machine formats and hypervisors to match them, and despite the desire for a common VM format, the virtual server stacks got siloed into ESXi, KVM, Xen, and Hyper-V stacks with some spicing of PowerVM, Solaris containers and LDOMs, and VM/ESA partitions sprinkled on.

  With containers, the consensus has been largely to support the Docker format that was inspired by the foundational Linux container work done by Google, and Docker, the company, was the early and enthusiastic proponent of its way of the Docker way of doing containers.

  Now, Docker …

  Read more

• Solving HPC Conflicts with Containers

  March 2, 2017 Ben Cotton

  

  It’s an unavoidable truth of information technology that the operators and users are sometimes at odds with each other.

  Countless stories, comics, and television shows have driven home two very unpleasant stereotypes: the angry, unhelpful system administrator who can’t wait to say “no!” to a user request, and the clueless, clumsy user always a keystroke away from taking down the entire infrastructure. There is a kernel of truth to them. While both resource providers and resource users may want the same end result — the successful completion of computational tasks — they have conflicting priorities when it comes to achieving …

  Read more

• Wrapping Kubernetes Around Applications Old And New

  February 16, 2017 Jeffrey Burt

  

  Kubernetes, the software container management system born out of Google, has seen its popularity in the datacenter soar in recent years as datacenter admins look to gain greater control of highly distributed computing environments and to take advantage of the advantages that virtualization, containers, and other technologies offer.

  Open sourced by Google three years ago, Kubernetes is derived from the Borg and Omega controllers that the search engine giant created for its own clusters and has become an important part of the management tool ecosystem that includes OpenStack, Mesos, and Docker Swarm. These all try to bring order to what …

  Read more

• Locking Down Docker To Open Up Enterprise Adoption

  February 10, 2017 Timothy Prickett Morgan

  

  It happens time and time again with any new technology. Coders create this new thing, it gets deployed as an experiment and, if it is an open source project, shared with the world. As its utility is realized, adoption suddenly spikes with the do-it-yourself crowd that is eager to solve a particular problem. And then, as more mainstream enterprises take an interest, the talk turns to security.

  It’s like being told to grow up by a grownup, to eat your vegetables. In fact, it isn’t like that at all. It is precisely that, and it is healthy for any technology …

  Read more

• A New Twist On Adding Data Persistence To Containers

  February 7, 2017 Jeffrey Burt

  

  Containers continue to gain momentum as organizations look for greater efficiencies and lower costs to run distributed applications in their increasingly virtualized datacenters as well as for improving their application development environments. As we have noted before, containers are becoming more common in the enterprise, though they still have a way to go before being fully embraced in high performance computing circles.

  There are myriad advantages to containers, from being able to spin them up much faster than virtual machine instances on hypervisors and to pack more containers than virtual machines on a host system to gaining efficiencies …

  Read more

• Riding The Coattails Of Google Kubernetes And AWS Lambda

  February 1, 2017 Timothy Prickett Morgan

  

  There are individuals and companies that create whole new technologies for their own consumption and that sometimes open source them for others to help steer their development and fix their bugs. And then there are still other companies that polish these tools, giving them some enterprise fit and finish, and thereby make it possible for others to deploy a particular technology without having to have PhDs, who are not available anyway, on staff.

  From the enterprise perspective, the Apache web server and related Tomcat application server needed its Big Blue, the Linux operating system needed its Red Hat, and the …

  Read more

• Looking Through the Windows at HPC OS Trends

  January 30, 2017 Ben Cotton

  

  High performance computing (HPC) is traditionally considered the domain of large, purpose built machines running some *nix operating system (predominantly Linux in recent years). Windows is given little, if any, consideration. Indeed, it has never accounted for even a full percent of the Top500 list. Some of this may be due to technical considerations: Linux can be custom built for optimum performance, including recompiling the kernel. It is also historically more amenable to headless administration, which is a critical factor when maintaining thousands of nodes.

  But at some point does the “Windows isn’t for high-performance computing” narrative become self-fulfilling? …

  Read more

• OpenHPC Pedal Put To The Compute Metal

  November 28, 2016 Timothy Prickett Morgan

  

  The ultimate success of any platform depends on the seamless integration of diverse components into a synergistic whole – well, as much as is possible in the real world – while at the same time being flexible enough to allow for components to be swapped out and replaced by others to suit personal preferences.

  Is OpenHPC, the open source software stack aimed at simulation and modeling workloads that was spearheaded by Intel a year ago, going to be the dominant and unifying platform for high performance computing? Will OpenHPC be analogous to the Linux distributions that grew up around …

  Read more

• Google Wants Kubernetes To Rule The World

  November 8, 2016 Timothy Prickett Morgan

  

  At some point, all of the big public cloud providers will have to eat their own dog food, as the parlance goes, and run their applications atop the cloudy version of their infrastructure that they sell to other people, not distinct and sometimes legacy systems that predate the ascent of their clouds. In this regard, none of the cloud providers are any different from any major enterprise or government agency that struggles with any kind of legacy system.

  Search engine and online advertising giant Google wants its Cloud Platform business to compete against Amazon Web Services and Microsoft Azure and …

  Read more

Previous Articles Next Articles