Lets talk storage….

Let's talk about the storage advancements going on in data centers, cloud and on our laptops.  Capacity and speeds continue to increase while prices relatively remain the same, thereby giving the consumer more bang for the buck.  This shouldn’t come as a surprise given the observation of Moore law.  It all translates to more options and decisions available to us as consumers in all areas, whether it's your datacenters, cloud or laptop. 

How can I scale my (datacenter) storage? Should I upgrade to flash storage? Can I move all my data to the cloud?  My Flash storage drives are slow?  What is this new flash memory (NVMe) and what kind of performance will I see? 

Let's face it, people like the notion of having full control of their storage.  If my data is sitting on my drives in my rack, I have full control over it and its safe.  While data is moving to the cloud there will always be a desire and an arguable need to have local storage in your datacenter, assuming you still have one.  There is a sense of security knowing where your drives are and who has access to them.

Server Storage

If you are interested in local server storage, chance are you might already have a server storage array.  We might ask how should I scale my storage?  "Scaling out" (or horizontal scaling) refers to adding more nodes to your storage network.  This can be done by adding another storage array to your storage pool or more simply, presenting the new storage to your compute resources like your hypervisors.  This method can be more costly but allows you to add both capacity and performance (controllers) to your storage resources. "Scaling up" (or vertically) means adding more resources to your existing storage.  This can be quick and cheap by adding more drives to your existing array and increasing your storage capacity.  There are advantages to both scaling up and scaling out and many storage solutions support both methods of scaling but your answer ultimately depends on existing hardware and your needs.

Should I upgrade my storage to a flash array?  Yes but don’t forget about your network.

If you can afford to, there are many performance gains offered by all flash arrays including raw hard drive speed as well as inline features made possible by flash like compression.  One important consideration when adding a much faster performing storage array is the medium that connects your storage resources to your compute resources. We must consider that storage network switch speed (1,10 or 40 Gb/s). If you are using a storage array connected via a gigabit switch (1Gbps link)  you will see read and write speeds of about 115MB/s  (coincidentally approaching the same theoretical speed of 1Gb/s = 1024Mb/8bits per byte = 128 MB/s¹).  If you upgrade your storage to an all flash solution with that same 1Gb/s network switch you will likely NOT see an increase in server write speed because your bottleneck will be your 1Gb/s network .    Yes there are other advantages that you can make use of on the back end but if you want faster write speeds to leverage your faster storage you will need a faster switch.   One of the reasons people might want flash server storage in your data center is for performance, just don’t forget about your network.

There is a flip side to this coin (local data center server storage) which is having the data is your full responsibility, you need to think about things like backing up your data (topic for another post), ensuring availability like power and connectivity, and well as managing security both physical and logical.  This is where cloud storage can alleviate some of these burdens.

Cloud Storage

Since storage is getting so cheap, do I want to bother with managing it? Why not pay someone to deal with all the network, drives, backups, (physical) security ² and just manage it for me?  There is no questions that this is a growing trend and for good reason.  There is comfort in knowing you have an entire team of storage, network, and security architects at Amazon, Microsoft or Google’s cloud that would be more than happy to manage your data and guarantee your data with umpteen 9's of durability and availability.  These Cloud Service Providers (CSPs) can geographically replicate your data all over the world and on their highly sophisticated content delivery networks (CDNs), making it always ready to your customers all starting at just a couple cents per GB.  Want your data encrypted at rest? Just check a box.  Compliance concerns? No problem, the cloud providers have a plethora of certifications and can generate your compliance reports on demand.

I don’t want to bother with scaling my storage or managing my storage network infrastructure, let's move it all to the cloud.

If you like, Amazon can ship you a product they call a snowball edge which is a storage server in a box that can store 100TB of data on 10, 25 or 40Gbps network link to help you move large quantities of data fast to the cloud.  If that is not enough space, Amazon has a datacenter packed in a truck they call the AWS Snowmobile that can store 100PB of data. Of course all the data is fully encrypted with 256-bit encryption keys.

Cloud solutions are truly starting to blur the line of local server storage and cloud storage offering storage gateways that can sit inside your datacenter or office and keep a local copy of your data and applications.  Microsoft StorSimple 8000 is a storage server device that sits on premise and can contain flash or magnetic storage over a fast iSCSI connection to store and replicate your data out to the Azure cloud.  Prefer a virtual appliance? No problem, Microsoft offers a StorSimple Virtual Appliance that can be deployed on your existing virtual infrastructure (Hyper-V or Vmware hypervisor). 

CSPs offer all the tools, benefits and capabilities to make your move to the cloud easy, convincing any skeptics that its worth the move. 

Enough about the servers...what is this new flash memory (NVMe) and what can it do for my laptop?

Computer (laptop/desktop) Storage

Flash storage has changed considerably in the last few years. Traditionally, we saw about a 4 times performance gain when upgrading to flash from magnetic storage.  A SATA magnetic drive might yield a read speed of about 150MB/s where as a SATA flash drive typically offers read speed of 550 MB/s.  There is a newer hard drive called NVMe (Non volatile memory express), that is roughly 5 times³ faster the traditional SSD storage and can offers read speeds of 3500MB/s.  How does it do this? It uses a faster connection to connect your hard drive to your computers motherboard.

Let's get technical….

NVMe take the traditional NVM, non-volatile memory, (retains the data when power is no longer being supplied to the storage) and adds the "express" notation for a new faster method of access. The "e" is the important part here, which allows the host hardware\software to make use of parallel access channels, up to 4 PCIe lanes to be exact.  In a nutshell, the traditional flash storage connects via a SATA connection that is limited to 6Gb/s where as the new NVMe storage connects to a PCIe port that can make use of 32Gb/s.  Note that I am focusing on the connections to the flash storage and not the storage itself. 

¹Security is a shared responsisiblity in the cloud.  While CSPs take care of the physical security for the most part, the consumer still  needs to take care of the logical security  e.g. Bring your own KMS, Transfering data over TLS 1.2, etc

²Important note about data transfer speeds.  The bit or Byte notation for all the speeds listed above are extremely important.  If you see a capital "B" that means Byte which is made up of 8 bits.  This means that 1Gb is equal to 128MB.  Often time network speeds are measured in bits like a 1Gbps network card and storage/drives are measured in "B" like the speed of the fastest consumer SSD drive available with a read speed of 3,500MB/s (3.4GB/s). Many people overlook the case of the “b” and it has a drastically different meaning.

³Careful if you decide to spring for one of these new NVMe drive because:

• Your laptop or desktop needs one of these M.2 connection to plug in this new drive (for a desktop you can buy a $15 adapter if your computer doesn’t come with a M.2 slot)

• Not all NVMe drive offer read or right speed that are close to the 32Gb/s (or 4,000MB/s) and many NVMe drives on the market are just as “Slow” or a little bit faster then traditional SATA SSD’s.  32GB/s read speeds is just the theoretical limit of the connection and right now (at the time of this post) the fastest available consumer NVMe drive seems to be a Samsung 960 Pro which writes at about 3500MB/s (or 28Gb/s). 

• Surprisingly this significant speed performance does not actually boost windows 10 boot times much over traditionally SATA SSDs however it does boost overall performance.  As a matter of fact, Intel released a “Optane Memory” which is essentially a NVMe drive used to cache program data and (Optane memory) cannot be enabled with NVMe hard drive since it is essentially the same thing.  The driver software will tell you that the Optane memory is not compatible with your configuration (because your boot drive is already an NVMe drive).

Like this article?  Share it and follow us on Facebook or LinkedIn