The cloud infrastructure market is estimated to be worth some $120bn, according to analyst house Forrester, and cloud storage accounts for a growing percentage of the enterprise storage market as it displaces on-premise arrays and appliances.
Within the push to cloud an emerging trend is for hybrid cloud storage. Hybrid architectures allow organisations to use the same storage infrastructure on-premise and in public cloud infrastructures. On paper at least, hybrid allows data to move seamlessly between cloud storage and local datacentres, and between cloud storage providers.
With the prevalence of object storage technology and the widespread adoption of Amazon Web Service’s (AWS’s) S3 application programming interfaces (APIs), hybrid storage has also gained ground among enterprises.
That trend accelerated with the need for organisations to provide remote workers with easier access to data during the pandemic.
“Traditional corporate file servers sitting behind firewalls, and VPNs [virtual private networks] are clunky to use when your workforce is mostly remote or spread across the world,” says Brent Ellis, a senior analyst at Forrester.
“Using a hybrid-cloud file server you can get files closer to the people that use them. You have an easy way to manage adding storage because you can use the cloud for expansion of capacity.”
Even so, there are obstacles to cloud storage. These vary from organisation to organisation, based on their application suite, IT maturity and workloads. These are some of the more common barriers, as well as some solutions.
Cloud is not suitable for all applications (or their storage)
By no means can all applications run in the cloud. This can be down to basic compatibility, or that the application will run, but the performance of cloud-based storage falls short of what the business requires.
Some applications require Posix-compliant storage to work. Conventional block or file storage systems are Posix compliant. Out of the box, object storage is not.
Cloud suppliers have been successful in moving file-based storage to the cloud, which works for unstructured data and applications such as archiving. Here, performance is within the bounds of what can be achieved with a connection to the cloud. But for high-performance applications that rely on block storage, there is still a gap.
“For file, there is a performance and management advantage to hybrid cloud storage, especially when dealing with a highly distributed workforce,” says Ellis.
“For object, hybrid solutions allow flexibility and ways to manage performance. For block, the desire is largely related to letting the application layer consume cloud resources and having storage that is flexible, but performance management must be architected.”
For block storage, on-premise storage still has the edge. Users can tune their arrays to suit their applications and install the exact levels of throughput and input/output operations per second (IOPS) they need. Databases, or database-based applications, work best with block storage.
The solution is to keep some workloads on-premise, and to put others – those that are less demanding – in the cloud. But technologies are emerging that allow workloads to move between cloud and on-premise systems.
One is Amazon S3 on Outposts, where AWS supplies on-premise S3 (object storage). Ellis also points to a Dell initiative, Project Alpine, which can move a Power Store volume between cloud and on-premise hardware.
“These sorts of hybrid systems need to be architected in datacentres that have high speed interconnects with the cloud to make that performance possible,” he says. “The advantage is in using compute or server-less functions from multiple clouds, and they need a way to abstract the storage across multiple environments, including on-premise.”
Hybrid cloud is more complex – and harder to manage
Hybrid storage systems can also create complexity. One area is the management layer, where some industry observers argue hybrid is unnecessarily complex compared to all-cloud or all on-premise systems.
At the very least, administrators face having to deal with two different management interfaces.
“To take advantage of some of the newer technologies, you quite often have to spend effort on transforming your applications so that they can take advantage of these new features,” says Tom Bragg, an IT infrastructure architect at consultants KPMG.
“That means that you have to find people who can still write applications you have to actually store all of the source code you have to… It’s not necessarily simple. And data storage is one of those things that’s always going to be a sticking point.”
The answer is to find technology that works, out of the box, across both environments. These include S3 Outposts, IBM Cloud Object Storage, NetApp’s StorageGrid, Cloudian and others. One factor in IT managers’ favour is that most use the AWS S3 APIs as building blocks.
“Hybrid and multicloud storage for S3 compliant storage is a no-brainer,” says Forrester’s Ellis. “Being able to have a consistent object storage layer for on-premise applications and cloud applications means that it is easier to move workloads to the cloud and back again.”
Hybrid cloud storage can cost more
Cloud storage is not always cheaper, and adding cloud capability to an existing storage system can add dramatically to costs.
Cloud’s variable cost structure offers advantages over the up-front cost of on-premise systems, but firms can incur fees both for long-term storage of data and for data egress.
Applications that burst to the cloud can incur unexpected data storage fees, as can bringing data back in house.
Firms can avoid the worst of these fees by modelling likely usage and costs, and by designing their cloud setup to minimise egress costs.
“People tend to put their data in a single cloud or keep it or single cloud for precisely those reasons. It costs a lot of money to move data around and to exfiltrate it every time,” says KPMG’s Bragg. At the same time, these costs can be offset through lower licensing and database admin expenses.
Cloud-native applications usually need object storage
One of the stronger drivers to move towards hybrid cloud and hybrid cloud storage is the trend towards cloud-native applications. Conventional applications can be hard to port to the cloud, but cloud technologies can be run on-premise.
This is still a developing area. But if an application or workflow can run in the cloud, it can also run on cloud architecture, on-premise or move between cloud providers.
“To make the most of hybrid and multicloud, organisations need to leverage the common denominators across their silos, and that means turning to Kubernetes and container-based applications which – once architected – can run in any cloud,” says Patrick Smith, field chief technology officer (CTO) of Europe, the Middle East and Africa (EMEA) at Pure Storage.
Containerisation does not solve the storage problem itself, but it gives developers a path to object storage. As Ellis points out, competition between vendors should make hybrid storage “easier to consume and cheaper”.
A hybrid future?
Whether containerisation will solve all the issues of hybrid architectures remains to be seen. But although most, if not all, businesses state that they want to move to the cloud, according to KPMG’s Bragg, they are not there yet. Hybrid is a useful bridge to that future.
“You’ve accepted that you’ve got a five- to 10-year roadmap on-premise, so you have to make a decision about whether you want to provide cloud-like services to your on-premise, hosted users,” he says.
“That’s a conversation that will include object storage. Do you want to provide that capability on-premise to allow people to put a stepping stone into the cloud?”