As containers take over the IT environment, a problem that has existed since virtual machines (VMs) became popular has become more urgent: how to keep persistent storage, and keep it consistent.
Just as a business designed for foot traffic of 50 to 100 people per hour does not work well when it sees 500 to 1,000 people per hour (think Black Friday in most places these days), storage designed to hold data that persists across instances and even installations often struggles in containerized environments.
There are a couple of reasons why this is so.
First, we now treat our servers as disposable resources. If one dies, we leave it dead and simply spin up another. Normal processing keeps the dead server for later evaluation, but many shops do not follow this best practice. The issue has to do with servers that die when writing to persistent storage—what was the state of the write? Did it get interrupted mid-transfer? Luckily, applications can be designed to account for this, and the increasing prevalence of transactional processing in applications that are not based on a database management system (DBMS) shows that people are doing just that.
Second, the number of server instances hitting those persistent stores is higher in a containerized environment—they are smaller, lighter and faster to spin up and their container density is generally higher than VM density. Contention issues go from zero to full-on when an app goes from “one server is writing to this persistent store” to “two servers are writing to this persistent store.” And often there are hundreds of servers writing to the persistent store. As mentioned, the stop-gap is to implement some form of transactional processing to prevent writes and overwrites.
Persistent Storage: It’s Got Issues
The problem is, we end up with transactional processing written into dozens or hundreds of server instances to protect a small number of persistent disks. This is not optimal, and at a minimum increases software complexity.
That’s why you’ll be hearing more. Here are a few places working on simplified but powerful persistence ideas:
{code} Rex Ray
Originally part of EMC, {code} is Dell’s open source group and it has led the charge on the container storage interface (CSI) that facilitates persistent storage. Examples the company has implemented include databases and Minecraft, but the technology works for anything that must be stored on disk.
Container Orchestration Engines and Cloud Providers
All of the well-known container orchestration engines (COE) and infrastructure-as-a-service (IaaS) cloud vendors have methods to handle persistent disk—some more transactional than others and some more work than others. Expect this process to become standardized (and possibly for that standardization to take the form of CSI above) in the next year or so. If persistent data is critical to your container environment, it is worth looking at how each solves the problem, as some of the solutions are procedural (such as Docker’s recommendation), and thus are somewhat portable across COE solutions. The weakness of IaaS providers is portability between systems, but that is true of all IaaS procedures/solutions at this time.
Virtualization Engines
Both VMware vSAN and Microsoft NTFS have systems for providing persistent disk. While much of this is the simple case of “disk that doesn’t disappear when the container does,” that is the bulk of persistence needs. Expect the support to continue to expand, perhaps even going so far as to implement 100 percent transactional shares.
Storage Array Vendors
Storage array vendors have useful tools for this area. While what you use to access their storage is not too terribly large a concern to these vendors, they do have a vested interest in seeing that their storage meets the needs of the changing enterprise. As such, they are delving into methods to help make persistent storage available and consistent. This is not a huge stretch for the vendors, as the systems they produced were always intended for multiple access. Where they lag (and where Rex Ray steps up) is in interoperability/interchangeability of vendor solutions.
Conclusion
For those with controlled access to persistent/shared storage, these issues are not that big a deal. For high-volume or high-turnover DevOps installations, though, persistent storage is another link in the chain of reliability that must be protected. Keep kicking it, and hopefully before your need becomes urgent, a solid system will be in place to manage and protect persistent stores.
- 3 Tips to Avoid Painting Your DevOps into a Scalability Corner, January 19, 2016
- Diamanti Accelerates Continuous Integration Pipeline at MemSQL with New Container Networking and Storage Capabilities, August 22, 2016
- Is Your Storage Too Slow for DevOps?, February 13, 2018
Enhancing OpenShift Security for Business Critical Deployments
