How to resolve this conflict? Many leading organizations today in business, science, and engineering are turning to a special computing infrastructure, rather than individual systems, to drive innovation and deliver breakthrough results. That special infrastructure—called high-performance computing or HPC—consists of a server cluster in which the member servers or nodes can actually combine their processing power to work on a single problem at the same time.
HPC clusters take the concept parallel processing and blow it up to the size of an entire data center. In doing so, HPC essentially turns multiple, off-the-shelf servers into one big supercomputer. The power of an HPC cluster increases proportionally with the addition of each node, and HPC clusters can scale as much as necessary. So, with the design of HPC allowing essentially unlimited processing power, processing speed can often be removed as the system bottleneck, even for complex problems in AI, big data, and engineering.
That’s especially true with many of the complex problems being handled today, which deal with large datasets. In such an environment, storage is likely to become the bottleneck, and data-transfer rates especially important to the speed of an overall system. And that’s where faster, innovative storage media, like Intel Optane technology, comes in. Intel Optane is Intel’s implementation of Intel 3D XPoint memory, a breakthrough storage technology that combines near-RAM speed with the persistence of normal storage drives. Intel Optane SSDs help unleash the power of HPC systems by improving input/output (I/O) rates beyond what has previously been possible in storage drives, effectively speeding up problem solving for data-driven AI, research, analysis, engineering, modeling, and other fields.
Let’s look at some of the use cases for Intel Optane SSDs within an HPC environment. First off, there’s a cool new technology called Intel Memory Drive Technology that essentially turns Intel Optane SSDs drives into additional system memory through the magic of software-defined memory (SDM). Nodes in an HPC system are often fitted with the maximum amount of system memory, but the number of slots is always limited. With Intel Memory Drive Technology, a system can expand its DRAM capacity by up to eight times, providing extremely low latencies and boosting the performance of individual nodes in the HPC cluster.[1]
Intel Optane SSDs can also improve the performance of HPC when they are treated normally as storage devices, as in the following examples:
- Temp storage: Intel Optane SSDs can improve performance for HPC workloads requiring large scratch/temp storage; for example, applications in genomics, fluid dynamics, computational chemistry, and structural analysis.
- I/O bursting: Intel Optane SSDs can be used across HPC nodes to accelerate data transfers and handle I/O spikes.
- Checkpoint/memory snapshots: Intel Optane SSDs can reduce the time required to restart a virtual machine (VM) after a checkpoint or memory snapshot.
- Parallel file system metadata: HPC requires a parallel file system, the most common of which is Lustre. Intel Optane SSDs help improve the performance of Lustre and other parallel file systems by improving the I/O of metadata operations.
HPC, Intel 3D XPoint technology, and Intel Optane SSDs are growing in importance as organizations look for ways to push boundaries in AI, big data, engineering, design, and other fields. These technologies work better together by reducing the processing and I/O bottlenecks within a system, enabling companies to engineer products and make breakthrough discoveries faster than ever.
At Prowess, we understand HPC and Intel Optane technology, not just in isolation, but as elements in a complete landscape that we view every day from our front-row seats to the tech world. Our understanding of this broader view makes us a great partner to clients who want tell stories about their technical products and how those products solve the most important challenges business face today.
To keep up with more emerging trends and technologies, follow Prowess on our blog, Twitter, and LinkedIn.
[1] Intel. “Intel® Memory Drive Technology.” January 2018. https://www.intel.com/content/dam/support/us/en/documents/memory-and-storage/intel-mdt-setup-guide.pdf.