Making Sense Of Sensu

Former British Prime Minister Harold Wilson said that a week is a long time in politics. What Wilson described was the rapid change of pace in the political landscape where the fortunes of any politician or political group could change dramatically in the course of a single week. Technology isn’t all that much different. It changes fast.

Here is a case in point. It has been six months since Sean Porter, chief technology officer of Sensu, mapped out the road ahead, and The Next Platform caught up with Porter prior to Chef Conf 2018 to find out what has been happening at the full stack monitoring startup.

The Sensu project has actually been open source for over six years. At the start, Sensu was incubated inside the company that Porter and Caleb Hailey, now chief executive officer at Sensu, had built called Heavy Water. The original company was a traditional PaaS– as in “People as a Service” – play. It was formed out of a team of DevOps experts who then made themselves available to consult and help other companies and organizations with their DevOps strategy. As is the case more often than not, when you boil away all the vapor on DevOps, it is actually monitoring, and monitoring increasingly complex environments was ultimately a key part of early DevOps use cases.

You can’t fix what you don’t know about, and Porter and Hailey quickly realized that the software they were creating coupled with smart people was going to play out better than the company’s people-only play. That’s the thing about software: once you write it, there is very minimal overhead, and distribution is just a simple matter of hitting the copy button. People don’t scale, but software does, it always has. So in January 2017, Heavy Water refactored its staff of nine folks and pivoted full time into the existing open source Sensu project to then build a business around it.

Start Making Sensu

By December of 2017, Sensu vice president of engineering, Greg Poirier, was talking up the company’s hiring practices and how it interviews engineers. By the time we spoke with Porter, he talked about having 34 people on the books – that’s a serious amount of hiring in a short time for this Portland, Oregon operation. Sensu has also recently been given a significant injection of $10 million from Battery Ventures, with participation from Foundry Group, to keep all this hiring momentum going.

CEO Hailey said in the statement about the Series A funding: “Sensu Inc will be 50 employees by the end of 2018. The infusion of new capital into Sensu will allow us to continue making investments in Engineering the future of Sensu.” This is a company that strongly believes in open source and community. As founding members of the popular Automacon series, Sensu members have proven that they were also able to bring together many diverse groups from across the DevOps community and break down barriers and have meaningful conversations. That is a rare skill.

Throwing It All Away

After all of this injection of effort, money, resources, and people, what did the Sensu team do? They threw away all their old software. That’s right, they literally refactored the entire thing. Why?

Jason Dixon recently spoke of version 2.0: “All throughout, our engineering team has been hard at work building Sensu 2.0, a new framework written in Go and designed from the ground up to be more portable, easier and faster to deploy, and (even more) friendly to containerized and serverless environments.” These are bold statements for a very young company, it takes some serious vision to throw away all your old stuff and start all over again. Undeterred, that was exactly what they did. They were not however completely reckless, Dixon added, “But above all, it was designed to support all of the features and functionality you’ve come to know and love about Sensu 1.0.”

In our conversation with Porter, we asked what really drove this bold refactoring, and why there was such a huge turn around. Porter explained: “What Sensu is at its heart, is an open source monitoring tool with a focus on composability and extensibility. We spent years finding out what works and what doesn’t.” Continuing to add that they wanted 2.0 to essentially automatically handle the “naturally messy environments” that modern hybrid computing brings.

Containers, Contain Thyself

Porter continued to explain that the number of ephemeral computing nodes is on the rise. Containers and ephemerality feature heavily in the company’s design decisions, especially around the concept of “monitoring the herd.” To solve for the thundering herd issue, Sensu in concert with the community built Kubernetes side cars. The challenge of ephemeral thundering herds of unmonitored compute is clearly front and center for Porter. Sensu split the monitoring application “sensu-agent” into two pieces, there is now an EVENT API and a STATSD interface so you can scrape either directly. Porter is also obsessed with making sure that the client code does not interfere or burden the actual container, he mentioned that “It’s no good if the monitoring software client is bigger than the software you are trying to contain.” This also explains their need to “go” with Go as a way to try to lighten the load of their client software.

In speaking with Porter, we were reminded of this new thin client and those “touchless car wash systems,” which are new-fangled systems that are designed to wash a vehicle without scratching the paintwork. The problem is because they never physically touch the vehicle, they never actually clean the car. There is the same challenge with operating system and systems monitoring – you have to actually inspect the system and “touch it” to be able to interrogate it sufficiently to know what is going on. Porter explained that many of their largest customers (listing GoDaddy, T-Mobile and GE as its big partners) use a variety of “touch settings.” Porter was keen to mention that some only ever use the STATSD component of the Sensu agent for more simple data collection. Trying to be all things to all people is the goal here.

We also got into the issue of database backend scalability in a world with “servers are cattle,” we even went so far to say that “servers are wheat” should you even care to want to monitor individual containers, or ears of wheat? Porter explained that the Sensu backend wasn’t designed to be the one backend to rule them all. Rather than provide a whole soup to nuts experience, the company focuses on only the most recent events and store those but then rely on other external logging and storage systems such as Graylog, Splunk or Elasticsearch systems like InfluxDB or Graphite. Sensu focuses on what the state of the system looks like right now, and store just enough context to be able to quickly grab service health checks and not replay all of the history.

Continuing this thread of a company that really wants to try to partner, Sensu is essentially channeling its early consulting roots, Sensu has an interesting one-to-one partner program it is calling an Accelerated Feedback Program (AFP), which Sensu hopes will help define the future of the product. Porter said about AFP: “Our Product and Engineering teams will work closely with you, gathering quantitative and qualitative feedback to understand your needs, tracking improvements in Sensu 2.0 as we work towards the GA release, and challenging our assumptions about what a monitoring framework really looks like.” Because this is offered up for no charge, Sensu has enough resources and measured constraint for this, it could be a large amount of work.

The second Sensu Summit will be held at the Portland Art Museum, the home of the Automacon series, on August 22 and early announced keynotes include Workday and Box.com.