Every IT organization wants a more scalable, programmable, and adaptable platform with real-time applications that can chew on ever-increasing amounts and types of data. And it would be nice if it could run in the cloud, too.
Because of this, companies no longer think about databases, but rather are building or buying data platforms that are based on industry-standard technologies, big data tools like NoSQL and unified in a single place. It is a trend that started gaining momentum around 2010 and will accelerate this year, according to Ravi Mayuram, senior vice president of engineering and chief technology officer at Couchbase, one company that is building such a data platform for others to buy.
The cloud is important because it gives databases elasticity beyond what enterprises can often put into their on-premises environments, and with the proliferation of industry-standard hardware, businesses can more easily scale their databases horizontally.
“Another thing that has gone along with this is moving away from a rigid schema to a flexible schema, because that’s what really liberates your data at the end of the day – the ability to scale as you need and the ability to adapt your application to your market need, which comes from the schema flexibility,” Mayuram tells The Next Platform. “At the highest level, from a database standpoint, these are the two main mega-trends in wanting to do more adaptive development and wanting to scale applications. It’s almost going with the sense of, ‘I don’t know big this thing is going to be or how big this application is going to be or how flexible this schema is going to be or what this offering of mine is going to be,’ and yet being able to adapt to that is where the industry has moved to, away from the earlier mindset of, ‘Let’s plan for everything to the T – procurement, provisioning, go take a number and come back to me after six months.’ Those days are gone. Applications evolve on an almost hourly basis, deploy on a weekly basis and scale on-demand. And with all of this, you need the always-on, five nines of availability, disaster recovery and global availability of data.”
Launched in 2011, Couchbase’s eponymous data platform is designed to offer enterprises a consolidated solution for data tasks from analytics to search to query with a consistent view of the data. It includes a scale-out, memory-first architecture, a core database engine, security and support for mobile and IoT. It leverages JSON for data exchange and N1QL (SQL for JSON) for faster development, in-memory replication, encryption, and other security capabilities, and cross datacenter replication to help keep data close to users through the cloud or in containers. The company has raised $154 million through six rounds of funding and includes Comcast, Disney, eBay, GE, Marriott, Cisco, and Verizon among its customers.
Enterprises are getting behind the concept of a single data platform that can perform the tasks they have always asked of their databases, and one that can adapt to changes in the business, scale to millions of users, is cloud-agnostic, and is cost effective, Mayuram says.
“This is happening,” he says. “This isn’t something new. But the way this it is happening now is cache here, search there, Hadoop system here, a SQL system there. It’s a pretty fragmented picture and that has taken its toll on the infrastructure tier for many of these companies. The trend is to move more and more away from these point solutions, to a platform that offers a cogent vision of how all of this can be solve in one platform, and how it can at the same time take care of your operational needs of being able to be cloud-friendly, elastically available, yet perform almost linearly for your first user, or your millionth user. All this has required a sort of rethink from the standpoint of, how has the underlying infrastructure truly changed?”
While memory was more expensive in 2017, there is a good chance prices will fall in 2018 and the general trend is obviously down over the long term. Networks are also more reliable and have a lot more bandwidth and pretty low latency. “So if you wanted to make these kinds of changes, how would you redesign your data platform? You can now start thinking of a memory-oriented or memory-first type architectures which are distributed systems that can elastically scale, and elastic scaling using commodity hardware. You just have to get the right type of hardware that you normally procure for your datacenter and you can keep maintaining them to scale.”
The need to adapt applications quickly has driven the shift from a rigid to a flexible schema, Mayuram says. In traditional environments, changing an application is a time-consuming process involving business logic, business objects, data columns and data rows.
“What you’re basically doing is taking this object and deconstructing it into tables and columns, and then you’re storing them, and then when you render them back into an application, you’re taking these tables and columns and forming them into a column,” he says. “All this translation is what delays the whole change process because it’s a very serious change management cost you undergo in order to make a database change and that has to be in sync with your application logic change.”
With single platforms like that from Couchbase, the business makes changes to the application logic and the object underneath it, and the data portion of the object is automatically put into the database, accelerate the process of changing the application. The database also scales by simply adding commodity hardware and boosting the processing and storage linearly. Such scalability is important at a time when databases are increasingly required to handles millions of users and interactions.
“The numbers of users that you handle, the concurrency that you handle, the performance are very different from traditional databases, so that’s where this move to horizontally scaling systems which are schema flexible comes into play,” Mayuram says.
Platforms going forward also will need to span the cloud to the network edge because of the rising importance of mobility in the enterprise. “Anybody that is building any application now has got to have a mobile story because that is where we consume,” he says. “If you are not thinking about that – how is this going to be available in your edge – it becomes another tangential thing that people are looking into that has its own set of issues because now you have to solve those problems not at the data tier, but the higher tier like the application tier, and there are point solutions, and the whole back-end of service kind of stuff becomes even more complicated.”
In an increasingly mobile world, data needs to be available at the edge of the network, close to the devices and the users themselves, but the device needs to be able to access data sitting on systems behind the firewall. Mobile applications are built with the idea that the application logic stays on the device, even while much of the data is stored elsewhere, and it’s through the application that the data is synced with the device.
How quickly enterprises adopt such concepts as data platforms remains to be seen, Mayuram says.
“There are digital transformation initiatives underway, but the leaders are the ones who truly understand that, when you talk about digital transformation, it’s all about data – how quickly can you get to data, how quickly can you sell it to you end user,” he says. “It’s all about engagement at the end of the day. The change has gone from thinking in terms of transactions to engagement. How quickly can you interact? That is the business transformation that is actually happening. Enterprises that understand this are investing and they’re making the change. Today they’re changing their infrastructure by bringing in the systems like this into their infrastructure and we’re seeing the acceleration of that.”