Big commercial and investment banks, hedge funds, stock peddlers, and other financial firms are in the business of making money from money, and when they do it right, they are successful because they know how to gather and process information in such a way that they manage risk. In some ways, financial firms are adventurous, and in other ways, they are risk averse, oddly enough.
And our 401(k) plans and savings accounts like it that way.
When it comes to their data processing and storage systems, financial firms are without a doubt among the most cautious on the planet, and not just because they are handling other people’s money as well as their own, but because of the tough regulatory environment within which they operate and the stiff penalties for not being able to show how and where their systems process and store data or for incorrectly securing that data. Making bets on investments is one thing, but making bets on the security of the data and algorithms that comprise those bets is a risk too far.
This, among other reasons, seems to be why the use of the public cloud has not exactly taken off among financial institutions. That was the message from the HPC for Wall Street – Cloud and Big Data event in New York City this week. While public cloud providers have come a long way is providing the kind of security and auditability that financial firms require, it looks like big banks, brokerages, and hedge funds still have to work with them to go the extra mile to ensure that virtual systems on the cloud are set up properly so they can host applications.
“One of the challenges that you have with any public cloud is how do you start to deal with the unknown unknowns,” explained Dino Vitale, who spoke on a panel relating to the use of public cloud compute and storage by large financial enterprises and who is director of distributed platforms at TD Securities. (Vitale has held similar positions at Citigroup, JP MorganChase, and Morgan Stanley.) “How do you deal with some of the potentials that may come down the pike that need to be expressed in your agreements? I call this doing the long division and a lot of the regulators will, for instance, check that you do encryption this way and do key management this way, and they will be happy with that. But you have to say show me the long division, show me everything in between. And typically, a lot of cloud providers are not used to this type of scrutiny.”
“We are in a period of low interest rates, and that means low volatility and that means there are not a lot of us not making a lot of money. People are looking for cost reduction, and are looking for much more efficient use of infrastructure and just-in-time capacity and paying for what you use resonates. That said, public cloud on Wall Street is still not a thing. It gets talked about, but I literally do not know anybody who is using a big public cloud.”
There are all sorts of data governance issues, adds Vitale. As an example, in certain geographies and jurisdictions, data can only be stored so far from the physical datacenter, and in others, there are precise requirements for the physical location of servers and storage. The issue becomes how much control do you actually have. When data is stored on the cloud, providers like Amazon Web Services, Microsoft Azure, and Google Compute Engine shard and replicate data across multiple regions and sometimes datacenters to ensure its durability and availability. But this may not necessarily be permitted by the regulations governing financial institutions. If you do your computing and storage internally, then you can control the flow of data in a way that is not always easy on a public cloud.
But that is not the only possible consumption model for financial services forms who want to shift work to the public cloud. There is always software as a service where the application provider is ensuring the security of the data and applications (not presumably but by default and as part of the service) instead of buying raw infrastructure that has to be made secure.
“Bloomberg you can consider to be a financial public cloud to some extent,” explained Harvey Stein, who is head of regulation and credit modeling at the financial services and media company. “But for the most part, we don’t run third party software. It is all internal software and access is via biometric systems and we have it locked down in ways that you don’t see on Amazon Web Services.”
Stein said that the financial services industry is starting to see the cloud used for algorithm testing. “There are two aspects of this when you look at it from a risk perspective,” Stein said. “One is whether the data is public or private and how accessible the data is. The flip side is that you are, say, a hedge fund and you are looking at investment strategies and your data is relatively public and but your algorithms itself to be very private. You need to make sure that nobody else has any access to it and nobody knows what you are doing. How do you keep your algorithm itself secret when you are running on it shared hardware?”
Another issue that financial services firms have to deal with that perhaps others do not is the reproducibility of results, said Vitale. Cloud hardware is virtualized and getting the same exact results for a computation on exactly the same hardware, which is sometimes something that big banks and brokers have to demonstrate, is not precisely possible. But you can do that on internal iron.
Yet another thing that financial services companies need to do is demonstrate that the public cloud is providing good value for the dollar compared to doing processing and storage in house. Financial services firms have to store data for seven years, and using the public cloud as a kind of cold storage can look attractive and frees up hardware in datacenters for other tasks, but it is not a simple thing even if you can lock down all the security and data governance issues.
“You can pay pennies per gigabyte for storage, but my caveat is that you have to know what your internal charge is – fully baked – to do a fair comparison. And the other caveat is that you have better understand your access patterns. The whole cloud paradigm is based on data retention with limited access.” In some cases, said Vitale, it is very expensive to move data, and then there is the issue when the eighth year rolls around and you need to delete data. How do you make sure it is deleted on the cloud? Do you trust Amazon, Microsoft, and Google, or do you trust and verify? And how do you do that when datasets are sharded and spread all over the place?
To get further perspective on the public cloud issues among financial services firms, The Next Platform rang up Jacob Loveless, the CEO at financial cloud provider Lucera, who said that “cloud” is basically a four-letter word in this market and that Lucera and its customers think of it as next generation hosting. Lucera runs containers on bare metal X86 servers and has a customized version of the SmartOS operating system created by Joyent that is itself based on Solaris Unix, created by Sun Microsystems and now controlled by Oracle. In addition, Lucera has created its own software-defined networking stack, which is used to link its datacenters in Chicago, Secaucus (outside of New York near where the stock exchanges are actually located), and London to the exchanges and to secure the links between machines and those exchanges.
“We are in a period of low interest rates, and that means low volatility and that means there are not a lot of us not making a lot of money,” Loveless tells The Next Platform. “People are looking for cost reduction, and are looking for much more efficient use of infrastructure and just-in-time capacity and paying for what you use resonates. That said, public cloud on Wall Street is still not a thing. It gets talked about, but I literally do not know anybody who is using a big public cloud – that might be because our customers tend to be more focused on the front office. But I think there are some problems that are really acute, and one of them is location. If you have got a workload that can really benefit from scale out economics, running on a few hundred machines, then you probably have to move a lot of data, and that data is going to be changing. Moving that data from, say, Secaucus, New Jersey to Ashburn, Virginia is going to take a while and it is going to be very expensive.”
Moreover, no one can afford to pay the virtualization overhead – in terms of capacity and latency – to run front office financial applications on the cloud. For those of you not up to speed on the financial services lingo, the back office processing are things like post trade risk analysis for securities and foreign exchange, transaction settlement and processing, regulatory and customer reporting – the kinds of work that is not latency sensitive and that tends to be done in big batches. The front office includes things like the trading systems, pre-trade risk engines, and real-time analytics and fraud detection, which are all latency sensitive. This is why Lucera has very zippy servers and storage and why it locates its facilities in the same CH2, NY4, and LD4 datacenters where a number of exchanges are also located.
Survey data backs up this sense that financial services firms are hesitant to embrace the big public clouds for their applications. Shagun Bali, and analyst at TABB Group, presented some data at the event that showed precisely what you might expect. According to the survey, 65 percent of those financial firms polled said that data security concerns were an inhibitor to investments in the public cloud, and all other issues – integration with legacy systems, lack of clarity on costs and ROI, internal resistance from the IT staff, and high monetary and reputational risk – all took a backseat by far. When asked about the perception of public cloud infrastructure, 29 percent said they were very uncomfortable with how the public cloud was perceived, 24 percent said they were uncomfortable, and another 24 percent said they were neutral. That left 14 percent who were comfortable and only 9 percent who were very comfortable.
That said, financial firms told TABB they were keeping an eye on the public cloud and were looking to it for scale workloads (64 percent), cut costs (55 percent), and improve performance (47 percent). As for what workloads were being deployed on public clouds, about 23 percent of the survey respondents were using public cloud infrastructure to store data, another 14 percent were doing analytics on clouds, 14 percent were doing algorithm development, and 8 percent were generating their regulatory reports.