Brian Jackson - Fotolia


NVDIMM and RDMA offer significant virtualization advantages

Due to their effect on virtualized clusters, virtual SANs and overall database performance, new in-memory technologies like DRAM and RDMA networks are becoming more widely adopted.

The introduction of NVDIMM can improve database performance, but the real value will be in virtualized clusters and to accelerate distributed virtual SANs.
Persistent Dynamic Random Access Memory -- also called Storage-Class Memory -- is a hot topic in the IT industry today. This in-memory technology has boosted database performance by as much as 100 times, leading to a wider adoption of the in-memory approach. Due to this, new techniques for sharing memory and reducing access latency to persistent storage have received additional attention.

NVDIMM types and use cases

One such method is to make Dynamic Random Access Memory (DRAM) itself persistent. There are several competing approaches to this. The fastest is to add a backup flash memory module to a standard Dual In-line Memory Module (DIMM), so that all data is backed up when power is removed or a failure occurs. Viking Memory, Netlist and Micron supply DIMM modules with up to 16 GB capacity. This type of memory, Non-Volatile Dual In-line Memory Module-N (NVDIMM-N) behaves exactly like DRAM in operation, so it is word-addressable and requires refresh of the DRAM space.

The second type, Non-Volatile Dual In-line Memory Module-F (NVDIMM-F), is basically a DIMM full of flash, and behaves more like a disk drive with block-IO. It is much slower than NVDIMM-N, but provides much greater storage capacity, into the terabyte range.

The two types of NVDIMM have somewhat different use cases. It is less difficult to add NVDIMM-F to a system, since it can be made into a mountable disk drive volume quite easily. As a result, it fits any application without a major rewrite. NVDIMM-N can also be made into a mountable volume through the use of a RAMDisk software module that separates it from the true DRAM space. In this mode, NVDIMM-N has nanoseconds of latency versus the microseconds of flash.

Because NVDIMM-N appears as just another DRAM space to the OS, it is possible to extend operation and dramatically improve performance over the block-IO access method, though some adjustments must be made to the software to get the most out of the product. The application, OS, compiler and link loader all have to recognize persistence. Intel's NVIDIA Management Library addresses many of these issues for Linux, and Intel and Microsoft are working together on a Windows offering.

The payback for all of these software changes is that the persistent memory can now be accessed on a word-by-word level, or possibly even byte-wise. This means writing bypasses the whole file-IO stack and occurs at instruction speed. Writing a word takes just one instruction cycle instead of thousands, and read-modify-write operations are no longer needed.

The power of this in databases is obvious and would be a way to make those 100 times improvements head into the stratosphere.

Sharing NVDIMM spaces with RDMA

Performance in the NVDIMM class brings up issues of data integrity and data sharing. The ancient method of moving drives between servers when a failure occurs doesn't cut it with DIMMs buried inside a machine, especially when they are in cloud clusters. Data integrity with NVDIMMs means either writing data into a removable solid-state drive (SSD) and losing performance, or transmitting a replica to another server via Ethernet.

For today's large virtualized cluster, the obvious answer is to share NVDIMM spaces using the fastest network possible. This means using Remote Direct Memory Access (RDMA) over Ethernet, which perfectly fits the memory paradigm of NVDIMM. Data can move directly between NVDIMM modules with low system overhead and low latency. Moreover, RDMA supports word and byte operations, though we would need to extend those capabilities to work with the changes described above for NVDIMM-N.

In the meantime, block-IO over RDMA allows us to build very fast virtual SANs and share persistent DRAM data throughout a cluster. The speed benefits could be substantial. IBM, SuperMicro and HPE have announced support for NVDIMM products, and others are expected to follow suit.

Future NVDIMM applications

The high speed of NVDIMM and virtual SAN offer opportunities that extend well beyond databases. In-memory operations can accelerate most applications, with consequent reductions in job time or in the number of servers needed for a given workload. As block-IO devices, they are faster than the fastest non-volatile memory express SSD and can be viewed either as super-fast drives or as DRAM capacity extenders.

As DRAM extenders, even the humble world of hypervisors and containers can be directly impacted by being able to pack in more instances. NVDIMM can also improve performance of local instance stores, even if the replica write needed for data integrity were made to, say, an all-flash array. The NVDIMM approach would also facilitate big data and in-memory database instances in both public and private clouds.

That brings up Intel/Micron's 3D X-Point. Announced in late 2015, this is an SSD technology and an NVDIMM product. This new type of memory promises much faster speed than flash, ranging from five to 10 times slower than DRAM itself. If it lives up to the hype, X-Point will take NVDIMM storage to a new level of usefulness.

NVDIMMs and RDMA networks are in the earliest stages of development, and we'll see a lot of change in hardware and software as they continue to evolve. Despite their novelty, the benefits they promise are so great that IT departments should start planning for them, since they'll likely play a significant role in mainstream computing within the next few years.

Next Steps

How does RDMA work with flash?

How to use flash to complement DRAM

A beginner's guide to in-memory databases

Dig Deeper on Virtualized clusters and high-performance computing