--- description: The world of Artificial Intelligence is evolving at breakneck speed, blink, and you will miss the next advance. With model sizes getting larger and... title: "Open The Wallet Bay Doors Hal: The Power of Multi-GPU Utilization and Model Parallelism" image: https://www.storagereview.com/wp-content/uploads/2023/05/StorageReview-HP-Z8-Fury-G5-09.jpg --- [![Storage Review]()](https://storagereview.com/) --- [Facebook](https://www.facebook.com/@storagereview) [X (Twitter)](https://twitter.twitter.com/storagereview) [Instagram](https://www.instagram.com/storagereview/) [LinkedIn](https://www.linkedin.com/company/storagereview-com) [YouTube](https://www.youtube.com/user/storagereview?sub_confirmation=1) [Email](mailto:info@storagereview.com) [Spotify](https://open.spotify.com/show/1y6VnznABhHeOSMOmbDTz0) [Reddit](https://www.reddit.com/r/StorageReview/) [Discord](https://discord.gg/TwMHb4azdC) [RSS Feed](https://www.storagereview.com/rss.xml) [TikTok](https://www.tiktok.com/@storagereview) [Facebook](https://www.facebook.com/@storagereview) [X (Twitter)](https://twitter.com/storagereview) [Instagram](https://www.instagram.com/storagereview/) [LinkedIn](https://www.linkedin.com/company/storagereview-com) [YouTube](https://www.youtube.com/user/storagereview?sub_confirmation=1) [Email](mailto:info@storagereview.com) [Spotify](https://open.spotify.com/show/1y6VnznABhHeOSMOmbDTz0) [Reddit](https://www.reddit.com/r/StorageReview/) [Discord](https://discord.gg/TwMHb4azdC) [RSS Feed](https://www.storagereview.com/rss.xml) [TikTok](https://www.tiktok.com/@storagereview) [![StorageReview.com]()](https://www.storagereview.com) ≡ Menu - [Home](https://www.storagereview.com/) - [Storage Reviews](https://www.storagereview.com/review) - [End User Reviews](https://www.storagereview.com/consumer) - [Enterprise Reviews](https://www.storagereview.com/enterprise) - [Ubiquiti Reviews](https://www.storagereview.com/best/ubiquiti-reviews) - [SR Merch](https://store.storagereview.com) - [Leaderboards](https://www.storagereview.com/best) - [Best Storage Arrays](https://www.storagereview.com/best/storage-arrays) - [Best Enterprise SSDs](https://www.storagereview.com/best/enterprise-ssds) - [Best Servers](https://www.storagereview.com/best/servers) - [Best Enterprise GPUs](https://www.storagereview.com/best/enterprise-gpus) - [Best Desktops for Local AI](https://www.storagereview.com/best/desktops-local-ai) - [Best Laptops for Local AI](https://www.storagereview.com/best/laptops-local-ai) - [Best Local LLM Tools](https://www.storagereview.com/best/local-llm-tools) - [Agentic AI Hardware](https://www.storagereview.com/best/agentic-ai-hardware) - [Best SSDs & Hard Drives](https://www.storagereview.com/best_drives) - [Best Portable SSDs](https://www.storagereview.com/best/portable-ssds) - [Best Business Laptops](https://www.storagereview.com/best/business-laptops) - [Best Mobile Workstations](https://www.storagereview.com/best/mobile-workstations) - [Best Desktop Workstations](https://www.storagereview.com/best/desktop-workstations) - [Best Mini PCs](https://www.storagereview.com/best/mini-pcs) - [Best Laptop Battery Life](https://www.storagereview.com/best/laptop-battery-life) - [Storage Reference Guide](https://www.storagereview.com/storage-reference-guide) - [About SR](https://www.storagereview.com/about-storagereview) - [StorageReview.com Sweepstakes Rules and Regulations](https://www.storagereview.com/storagereview-com-sweepstakes-rules-and-regulations) Search [Home](https://www.storagereview.com/) » [Review](https://www.storagereview.com/review) » Open The Wallet Bay Doors Hal: The Power of Multi-GPU Utilization and Model Parallelism # Open The Wallet Bay Doors Hal: The Power of Multi-GPU Utilization and Model Parallelism by Jordan Ranous on June 16, 2023 [AI](https://www.storagereview.com/enterprise/ai)  ◇  [Enterprise](https://www.storagereview.com/enterprise) The world of Artificial Intelligence is evolving at breakneck speed, blink, and you will miss the next advance. With model sizes getting larger and larger, researchers and developers are constantly seeking ways to improve the efficiency and performance of AI models. One of the easiest ways to achieve this is using multiple Graphics Processing Units (GPUs) or Tensor Processing Units (TPUs, more on this in the next installment) for AI training and inferencing. ![]() DNN Training on the HP z8 G5 Fury Building on our [ last installment of AI In the lab](https://www.storagereview.com/review/the-storage-review-bare-bones-ai-setup-guide), we took a deep dive and practical hands-on look into the benefits of transitioning from using a single GPU to employing two, and eventually four, of these powerful cards in our HP Z8 G5 Fury workstation, with a specific focus on PyTorch model parallelism. ## The Power of Model Parallelism Before we dive into the specifics, it’s essential to understand the concept of parallelism. In the context of AI, parallelism refers to the process of running multiple computations simultaneously. This is particularly beneficial in AI training and inferencing, where large amounts of data need to be processed. PyTorch, an open-source machine learning library that we employ in the lab, offers model parallelism, which allows for the distribution of an AI model across multiple GPUs. This leads to faster training times, more efficient inferencing, and the ability to run larger, more complex models. ![]() Ensuring SLI is disabled is critical ## Benefits of Scaling Up ##### Single GPU Starting with a single GPU, this setup provides a solid foundation for AI training and inferencing. Running a single modern (or even a few generations old) GPU in a workstation for development is more than sufficient for the POC stage. It’s capable of handling a reasonable amount of data and can deliver satisfactory results for smaller AI models. However, as the complexity and size of the models increase, a single GPU could quickly struggle to keep up, leading to longer training times and slower inferencing. ![]() Single GPU Utilization ##### Two GPUs Switching to a pair of GPUs can notably amp up the performance of AI models. Think about it: twice the processing power can dramatically cut down training times, paving the way for quicker iterations and a swift journey to results. The inference stage also benefits, growing more efficient and capable of processing larger data batches simultaneously. In such an environment, PyTorch’s model parallelism comes into play. It effectively distributes the workload between the two units, maximizing their use. It’s a smart way of ensuring each piece of hardware carries its weight to achieve a highly productive AI operation. ![HP Z8 Fury G5 with Nvidia RTX A6000]() 3x the fun, NVIDIA A6000 ##### Four GPUs Scaling up to four GPUs takes the benefits of multi-GPU utilization to another level. With quadruple the processing power, AI models can be trained and inferred at unprecedented speeds. This setup is particularly beneficial for large, complex models that require substantial computational resources. PyTorch’s model parallelism can distribute the model across all four units, ensuring optimal utilization and performance. ![]() In a workstation, applying manual fan and clock values can yield training performance increases too. #### Implementation In The Lab Progressing from a solo unit to a duo and eventually to a quartet of GPUs for AI training and inferencing can unlock considerable advantages. Thanks to PyTorch’s model parallelism, these benefits can be optimally harnessed, yielding quicker and more efficient AI models. ![]() Trial and error, patience is key with AI/ML/DL Training. As our thirst for more intricate and competent AI swells, the adoption of multiple GPUs will undoubtedly grow in significance. In the next article, we will showcase the complexity improvements as you add more processing power and distribute across systems. Note: This article is based on the current state of AI and PyTorch as of June 2023. For the most up-to-date information, make sure to check our [most recent AI articles](https://www.storagereview.com/enterprise/ai). **Engage with StorageReview** [Newsletter](https://www.storagereview.com/storage_newsletter) |  [YouTube](https://www.youtube.com/user/StorageReview "Opens in a new window") | Podcast  [iTunes](https://podcasts.apple.com/gb/podcast/storagereview-com-storage-reviews/id1060681115 "Opens in a new window")/ [Spotify](https://open.spotify.com/show/1y6VnznABhHeOSMOmbDTz0 "Opens in a new window") |  [Instagram](https://www.instagram.com/storagereview/ "Opens in a new window") |  [Twitter](https://twitter.com/storagereview "Opens in a new window") |  [TikTok](https://www.tiktok.com/@storagereview? "Opens in a new window") |  [RSS Feed](https://www.storagereview.com/rss.xml) ![]() ### Jordan Ranous AI Specialist; navigating you through the world of Enterprise AI. Writer and Analyst for Storage Review, coming from a background of Financial Big Data Analytics, Datacenter Ops/DevOps, and CX Analytics. Pilot, Astrophotographer, LTO Tape Guru, and Battery/Solar Enthusiast. Previous post: [Dell OptiPlex Micro Plus Review](https://www.storagereview.com/review/dell-optiplex-micro-plus-review) Next post: [Dell Micro 815 Modular Data Center Brings Liquid Cooling to Extreme Edge](https://www.storagereview.com/review/dell-micro-815-modular-data-center-brings-liquid-cooling-to-extreme-edge) Trusted Vendors Products and solutions from our affiliate partners: - [ ![Ubiquiti Logo]() Ubiquiti ](https://store.ui.com/us/en?a_aid=StorageReview "Opens in a new window") - [ ![Newegg Logo]() Newegg ](https://click.linksynergy.com/fs-bin/click?id=g5terNMDhz0&offerid=1207190.18&subid=0&type=4 "Opens in a new window") - [ ![Amazon Logo]() Amazon ](https://amzn.to/4fLJUEW "Opens in a new window") Newsletter Subscribe to the StorageReview newsletter to stay up to date on the latest news and reviews. We promise no spam! 1 Leave this field empty if you’re human: ## Advertisement Content Categories [Facebook](https://www.facebook.com/@storagereview) [X](https://twitter.com/storagereview) [Instagram](https://www.instagram.com/storagereview/) [LinkedIn](https://www.linkedin.com/company/storagereview-com) [YouTube](https://www.youtube.com/user/storagereview?sub_confirmation=1) [Email](mailto:info@storagereview.com) [Spotify](https://open.spotify.com/show/1y6VnznABhHeOSMOmbDTz0) [Reddit](https://www.reddit.com/r/StorageReview/) [Discord](https://discord.gg/TwMHb4azdC) [RSS](https://www.storagereview.com/rss.xml) [TikTok](https://www.tiktok.com/@storagereview) Copyright © 1998-2025 Flying Pig Ventures, LLC Cincinnati, Ohio. All rights reserved. Manage your privacy To provide the best experiences, we and our partners use technologies like cookies to store and/or access device information. Consenting to these technologies will allow us and our partners to process personal data such as browsing behavior or unique IDs on this site and show (non-) personalized ads. Not consenting or withdrawing consent, may adversely affect certain features and functions. Click below to consent to the above or make granular choices. Your choices will be applied to this site only. You can change your settings at any time, including withdrawing your consent, by using the toggles on the Cookie Policy, or by clicking on the manage consent button at the bottom of the screen. Functional Functional Always active The technical storage or access is strictly necessary for the legitimate purpose of enabling the use of a specific service explicitly requested by the subscriber or user, or for the sole purpose of carrying out the transmission of a communication over an electronic communications network. Preferences Preferences The technical storage or access is necessary for the legitimate purpose of storing preferences that are not requested by the subscriber or user. Statistics Statistics The technical storage or access that is used exclusively for statistical purposes. The technical storage or access that is used exclusively for anonymous statistical purposes. Without a subpoena, voluntary compliance on the part of your Internet Service Provider, or additional records from a third party, information stored or retrieved for this purpose alone cannot usually be used to identify you. Marketing Marketing The technical storage or access is required to create user profiles to send advertising, or to track the user on a website or across several websites for similar marketing purposes. Statistics Marketing Features Always active Special features Special purposes Always active - Manage options - Manage services - Manage {vendor\_count} vendors - [Read more about these purposes](https://cookiedatabase.org/tcf/purposes/) Accept Deny Manage options Save preferences Manage options - {title} - {title} - {title} Manage your privacy To provide the best experiences, we use technologies like cookies to store and/or access device information. Consenting to these technologies will allow us to process data such as browsing behavior or unique IDs on this site. Not consenting or withdrawing consent, may adversely affect certain features and functions. Functional Functional Always active The technical storage or access is strictly necessary for the legitimate purpose of enabling the use of a specific service explicitly requested by the subscriber or user, or for the sole purpose of carrying out the transmission of a communication over an electronic communications network. Preferences Preferences The technical storage or access is necessary for the legitimate purpose of storing preferences that are not requested by the subscriber or user. Statistics Statistics The technical storage or access that is used exclusively for statistical purposes. The technical storage or access that is used exclusively for anonymous statistical purposes. Without a subpoena, voluntary compliance on the part of your Internet Service Provider, or additional records from a third party, information stored or retrieved for this purpose alone cannot usually be used to identify you. Marketing Marketing The technical storage or access is required to create user profiles to send advertising, or to track the user on a website or across several websites for similar marketing purposes. Statistics Marketing Features Always active Special features Special purposes Always active - Manage options - Manage services - Manage {vendor\_count} vendors - [Read more about these purposes](https://cookiedatabase.org/tcf/purposes/) Accept Deny Manage options Save preferences Manage options - {title} - {title} - {title} Manage consent Manage consent ```json {"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/www.storagereview.com\/review\/open-the-wallet-bay-doors-hal-the-power-of-multi-gpu-utilization-and-model-parallelism","url":"https:\/\/www.storagereview.com\/review\/open-the-wallet-bay-doors-hal-the-power-of-multi-gpu-utilization-and-model-parallelism","name":"Open The Wallet Bay Doors Hal: The Power of Multi-GPU Utilization and Model Parallelism - StorageReview.com","isPartOf":{"@id":"https:\/\/www.storagereview.com\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.storagereview.com\/review\/open-the-wallet-bay-doors-hal-the-power-of-multi-gpu-utilization-and-model-parallelism#primaryimage"},"image":{"@id":"https:\/\/www.storagereview.com\/review\/open-the-wallet-bay-doors-hal-the-power-of-multi-gpu-utilization-and-model-parallelism#primaryimage"},"thumbnailUrl":"https:\/\/www.storagereview.com\/wp-content\/uploads\/2023\/05\/StorageReview-HP-Z8-Fury-G5-09.jpg","datePublished":"2023-06-16T15:36:46+00:00","breadcrumb":{"@id":"https:\/\/www.storagereview.com\/review\/open-the-wallet-bay-doors-hal-the-power-of-multi-gpu-utilization-and-model-parallelism#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.storagereview.com\/review\/open-the-wallet-bay-doors-hal-the-power-of-multi-gpu-utilization-and-model-parallelism"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.storagereview.com\/review\/open-the-wallet-bay-doors-hal-the-power-of-multi-gpu-utilization-and-model-parallelism#primaryimage","url":"https:\/\/www.storagereview.com\/wp-content\/uploads\/2023\/05\/StorageReview-HP-Z8-Fury-G5-09.jpg","contentUrl":"https:\/\/www.storagereview.com\/wp-content\/uploads\/2023\/05\/StorageReview-HP-Z8-Fury-G5-09.jpg","width":1500,"height":1123,"caption":"HP Z8 Fury G5 Interior"},{"@type":"BreadcrumbList","@id":"https:\/\/www.storagereview.com\/review\/open-the-wallet-bay-doors-hal-the-power-of-multi-gpu-utilization-and-model-parallelism#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.storagereview.com\/"},{"@type":"ListItem","position":2,"name":"Review","item":"https:\/\/www.storagereview.com\/review"},{"@type":"ListItem","position":3,"name":"Open The Wallet Bay Doors Hal: The Power of Multi-GPU Utilization and Model Parallelism"}]},{"@type":"WebSite","@id":"https:\/\/www.storagereview.com\/#website","url":"https:\/\/www.storagereview.com\/","name":"StorageReview.com","description":"StorageReview.com is a leading provider of news and reviews throughout the entire IT stack - from the datacenter to the edge, and all points in between.","publisher":{"@id":"https:\/\/www.storagereview.com\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.storagereview.com\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.storagereview.com\/#organization","name":"StorageReview.com","url":"https:\/\/www.storagereview.com\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.storagereview.com\/#\/schema\/logo\/image\/","url":"https:\/\/www.storagereview.com\/wp-content\/uploads\/2020\/02\/Storage-Reviews-2-2.png","contentUrl":"https:\/\/www.storagereview.com\/wp-content\/uploads\/2020\/02\/Storage-Reviews-2-2.png","width":344,"height":61,"caption":"StorageReview.com"},"image":{"@id":"https:\/\/www.storagereview.com\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/x.com\/storagereview","http:\/\/youtube.com\/user\/storagereview"]}]} ```