AI workloads demand extreme data pipeline performance. Learn how HPE Solutions with WEKA are architected to deliver that breakthrough performance.
–By Don Wilson, WEKA Partner Manager, HPE
With the shift toward data-first modernization, companies are anchoring their business strategies on data to gain a competitive advantage. From commercial enterprises to research organizations to public institutions, artificial intelligence (AI) is emerging as the key workload for maximizing data leverage.
AI-driven workloads are projected to grow at an 18% CAGR through 2025, with enterprises deploying them at an aggressive rate.* As generative AI, machine learning (ML), deep learning, and technical computing workloads move to the mainstream, your business and IT infrastructure must keep pace with new high-demand performance requirements
AI workloads bring new challenges
Typically, AI-driven workloads have different stages in a data pipeline that introduce very different I/O profiles. They also require the support of many small, random reads simultaneously against massively scalable data sets, high read and write throughput, high read and write IOPs and low latency. The different stages of AI workflows also have varying performance, scalability, availability, flexibility, metadata handling, and data management requirements. In response, enterprises end up using multiple storage siloes, copying data between them. This is inefficient and also can lead to delays in data processing, which can significantly reduce time to insight and value.
Until recently, much of the focus for AI has been on model development and GPU deployment, at times neglecting the importance of feeding data-hungry GPUs with sufficient data to train models. GPUs can become a wasted investment if left idle, waiting for data. The faster data reaches the GPUs, the faster you’ll get your business outcomes. For example, faster data performance can translate to faster insights. Faster insights allow your teams to react faster to trends, significantly improving customer experience and creating new revenue opportunities.
AI demands performance at scale that cannot be addressed with legacy storage
Legacy storage and file systems can become a bottleneck in data pipelines. Conversely, traditional distributed and parallel file systems are typically complex, requiring a team of expert admins to configure, adjust and maintain the solution. You need a holistic data platform approach to harness massive machine learning and deep learning datasets to deliver the performance and scale required to use them efficiently and effectively.
This is where HPE Solutions with WEKA enter the picture.
Don’t allow inadequate performance to become an impediment to fully utilizing AI
HPE Solutions with WEKA are built on the WEKA® Data Platform’s advanced architecture, which leverages NVMe-based flash storage and ultra-low latency, high-speed networking to deliver orders of magnitude improved performance for AI/ML and analytics workloads as well as technical computing applications that leverage high-performance computing (HPC) and GPU servers. HPE Solutions with WEKA take full advantage of the latest flash, networking, and computing technologies to provide customers with the highest-performing solution for AI, ML and HPC.
HPE Solutions with WEKA include the HPE Alletra 4110 data storage server, which complements the WEKA® Data Platform and is designed to run the most performance-demanding, data-centric workloads from data stores for machine learning, deep learning, and analytics. It achieves this by delivering an ideal balance of superior data capacity, throughput, IOPS, latency, and processing for performance-intensive workloads.
The symmetrical system design of the HPE Alletra 4110 data storage server evenly spreads resources across processors, configuring the right mix of resources to meet the workload requirements.
It boasts a datacenter resource efficient 1U standard rack depth design, supporting up to 20 new generation Enterprise and Data Center Standard Form Factor (EDSFF) NVMe SSDs to which up to 315GB/s of PCIe Gen5 bandwidth, providing the highest performing interconnect for a variety of peripherals. It supports multiple high-performance networking cards with as much as 800Gbps of raw networking bandwidth through the system, supporting both InfiniBand and Ethernet connectivity. HPE Alletra 4110 eliminates the costly compromises of storage appliances and general-purpose server infrastructure.
Performance without scale is an AI showstopper
Meeting capacity requirements is an important variable when evaluating potential performance. Machine Learning workloads often achieve better results from large amounts of training data. Fully harnessing the benefit of that data requires planning to enable performance at scale. Systems that perform adequately with smaller datasets (e.g., a few hundred terabytes) may not perform nearly as well when datasets reach petabyte capacity in the future.
The WEKA® Data Platform provides more than just speed – it also provides a highly scalable, POSIX-compliant file system. WEKA clusters built on the HPE Alletra 4110 data storage server can start as small as 46TB and scale to hundreds of petabytes in a single namespace.
WEKA® Data Platform is designed to transcend the limitations of legacy file systems that would leverage local storage, NFS, or block storage, making it ideal for data-intensive AI and HPC workloads. The performance of HPE Solutions with WEKA scales linearly as more servers are added to the storage cluster, so it can easily scale with the increasing business demands.
HPE Solutions with WEKA software deliver flexibility, accessibility, and ease of use
When consolidating storage systems intended to equip AI workloads and data pipelines, the data platform needs to support a wide spectrum of I/O profiles and workload requirements and different access methods since applications will require a variety of access protocols. The WEKA® Data Platform supports NFS, SMB, S3, and GPUDirect Storage and provides its own POSIX-compliant intelligent client while simultaneously accessing the same global namespace. A CSI plug-in for Kubernetes is also available for container-based, high-performance workload requirements.
In addition, the WEKA® Data Platform’s zero-tune architecture doesn’t require special configurations to support different data and performance profiles, making it dramatically easier to manage. HPE Solutions with WEKA offer not only ease of use but also simple scaling and seamless sharing of data in virtually any location, delivered through a single storage architecture that runs on-premises or in the hybrid cloud with the performance of all-flash arrays while also enabling the tiering of data to an external S3 Object Store to provide greater flexibility and improved economics.
Rest at ease knowing HPE Solutions with WEKA are fully tested and validated to work together
HPE and WEKA have developed and validated an optimized HPE Solution with WEKA based on the HPE Alletra 4110 data storage server and WEKA® Data Platform to deliver the highest possible performance at scale. In addition to the HPE Alletra 4110 data storage server, HPE also offers purpose-built systems for WEKA based on HPE ProLiant Gen10 Plus systems, including the HPE ProLiant DL325 Gen10 Plus V2 and the HPE ProLiant DL360 Gen10 Plus. The HPE ProLiant DL family of servers are flexible, reliable, and performance-optimized rack servers that also support flash NVMe, making them another potential platform that can be used to support WEKA. The HPE ProLiant Gen10 Plus systems offer flexible options, a versatile design, and increased levels of security.
A cloud experience delivered as a service
HPE Solutions with WEKA may be purchased and consumed as a service via the HPE GreenLake edge-to-cloud platform. HPE GreenLake for Compute Ops Management is provided standard with the HPE Alletra 4110. Depending on your business needs, this allows you to invest via traditional ownership or pay-per-use models. These new solutions reinforce HPE’s strategy of delivering a cloud operational experience that is simplified, automated, and spans edge to cloud.
“The new HPE Alletra 4110 data storage server, powered by WEKA’s industry-leading data platform software, is an excellent example of the innovation HPE and WEKA are delivering together for customers”, says Jeff Echols, Vice President of Channel and Strategic Partnerships at WEKA. “This new solution leverages best-in-class technologies like PCIe Gen 5 connected NVMe and low latency high-speed networking to deliver exceptional storage performance and capacity density in a more energy efficient package. It also reflects the value of a validated end-to-end solution that HPE and WEKA customers have come to depend on for the leading-edge capabilities, and we believe they will appreciate the solution’s performance, density scalability, and sustainability benefits.”
