--- description: Large language models like GPT-4, meta LLaMa, and Alpaca pushing the boundaries of natural language processing and hardware to run them. title: Meta LLaMa And Alpacas Loose in the Lab! Running Large Language Models Locally image: https://www.storagereview.com/wp-content/uploads/2023/04/dual-rtx8000.jpg --- [![Storage Review]()](https://storagereview.com/) --- [Facebook](https://www.facebook.com/@storagereview) [X (Twitter)](https://twitter.twitter.com/storagereview) [Instagram](https://www.instagram.com/storagereview/) [LinkedIn](https://www.linkedin.com/company/storagereview-com) [YouTube](https://www.youtube.com/user/storagereview?sub_confirmation=1) [Email](mailto:info@storagereview.com) [Spotify](https://open.spotify.com/show/1y6VnznABhHeOSMOmbDTz0) [Reddit](https://www.reddit.com/r/StorageReview/) [Discord](https://discord.gg/TwMHb4azdC) [RSS Feed](https://www.storagereview.com/rss.xml) [TikTok](https://www.tiktok.com/@storagereview) [Facebook](https://www.facebook.com/@storagereview) [X (Twitter)](https://twitter.com/storagereview) [Instagram](https://www.instagram.com/storagereview/) [LinkedIn](https://www.linkedin.com/company/storagereview-com) [YouTube](https://www.youtube.com/user/storagereview?sub_confirmation=1) [Email](mailto:info@storagereview.com) [Spotify](https://open.spotify.com/show/1y6VnznABhHeOSMOmbDTz0) [Reddit](https://www.reddit.com/r/StorageReview/) [Discord](https://discord.gg/TwMHb4azdC) [RSS Feed](https://www.storagereview.com/rss.xml) [TikTok](https://www.tiktok.com/@storagereview) [![StorageReview.com](https://www.storagereview.com/wp-content/uploads/2026/07/Storage-Review-2x.png)](https://www.storagereview.com) ≡ Menu - [Home](https://www.storagereview.com/) - [Storage Reviews](https://www.storagereview.com/review) - [End User Reviews](https://www.storagereview.com/consumer) - [Enterprise Reviews](https://www.storagereview.com/enterprise) - [Ubiquiti Reviews](https://www.storagereview.com/best/ubiquiti-reviews) - [SR Merch](https://store.storagereview.com) - [Leaderboards](https://www.storagereview.com/best) - [Best Storage Arrays](https://www.storagereview.com/best/storage-arrays) - [Best Enterprise SSDs](https://www.storagereview.com/best/enterprise-ssds) - [Best Servers](https://www.storagereview.com/best/servers) - [Best Enterprise GPUs](https://www.storagereview.com/best/enterprise-gpus) - [Best Desktops for Local AI](https://www.storagereview.com/best/desktops-local-ai) - [Best Laptops for Local AI](https://www.storagereview.com/best/laptops-local-ai) - [Best Local LLM Tools](https://www.storagereview.com/best/local-llm-tools) - [Agentic AI Hardware](https://www.storagereview.com/best/agentic-ai-hardware) - [Best SSDs & Hard Drives](https://www.storagereview.com/best_drives) - [Best Portable SSDs](https://www.storagereview.com/best/portable-ssds) - [Best Business Laptops](https://www.storagereview.com/best/business-laptops) - [Best Mobile Workstations](https://www.storagereview.com/best/mobile-workstations) - [Best Desktop Workstations](https://www.storagereview.com/best/desktop-workstations) - [Best Mini PCs](https://www.storagereview.com/best/mini-pcs) - [Best Laptop Battery Life](https://www.storagereview.com/best/laptop-battery-life) - [Storage Reference Guide](https://www.storagereview.com/storage-reference-guide) - [About SR](https://www.storagereview.com/about-storagereview) - [StorageReview.com Sweepstakes Rules and Regulations](https://www.storagereview.com/storagereview-com-sweepstakes-rules-and-regulations) Search [Home](https://www.storagereview.com/) » [Review](https://www.storagereview.com/review) » Meta LLaMa And Alpacas Loose in the Lab! Running Large Language Models Locally # Meta LLaMa And Alpacas Loose in the Lab! Running Large Language Models Locally by Jordan Ranous on April 19, 2023 [AI](https://www.storagereview.com/enterprise/ai)  ◇  [Enterprise](https://www.storagereview.com/enterprise) In recent months, large language models have been the subject of extensive research and development, with state-of-the-art models like GPT-4, Meta LLaMa, and Alpaca pushing the boundaries of natural language processing and the hardware required to run them. Running inference on these models can be computationally challenging, requiring powerful hardware to deliver real-time results. In the Storage Review lab, we put two NVIDIA RTX 8000 GPUs on the job of running Meta’s LLaMa model to see how they performed when running inference on large language models. We used the [Lenovo P920](https://www.storagereview.com/review/lenovo-thinkstation-p920-tower-workstation-review-quadro-rtx-8000) as a host for the cards, minimizing bottlenecks. The RTX 8000 is a high-end graphics card capable of being used in AI and deep learning applications, and we specifically chose these out of the stack thanks to the 48GB of GDDR6 memory and 4608 CUDA cores on each card, and also [Kevin is hoarding all the A6000](https://www.youtube.com/shorts/9aoz_dAZnx0)‘s. ![meta llama NVIDIA RTX 8000 GPUs](https://www.storagereview.com/wp-content/uploads/2023/04/SR-dual-rtx8000.jpg) Meta LLaMA is a large-scale language model trained on a diverse set of internet text. It is publicly available and provides state-of-the-art results in various natural language processing tasks. In this article, we will provide a step-by-step guide on how we set up and ran LLaMA inference on NVIDIA GPUs, this is not guaranteed to work for everyone. ## Meta LLaMa Requirements Before we start, we need to make sure we have the following requirements installed: - NVIDIA GPU(s) with a minimum of 16GB of VRAM - NVIDIA drivers installed (at least version 440.33) - CUDA Toolkit installed (at least version 10.1) - Anaconda installed - PyTorch installed (at least version 1.7.1) - Disk Space; the entire set of LLaMa checkpoints is pushing over 200TB. Note: It is recommended to have a dedicated Anaconda environment for LLaMA, we have a few on the test bench so we can mess with packages and parameters and not hose the base installation. ![]() Set up a dedicated environment or two in Anaconda ## Steps to Run Meta LLaMA Inference on NVIDIA GPUs 1. **Download the LLaMA repository:** The first step is to clone the LLaMA repository from GitHub. `git clone https://github.com/EleutherAI/LLaMA.git ` 2. **Get the LLaMA checkpoint and tokenizer:** Visit the GitHub page, and fill out the linked Google form to access the download. Other LLMs are available, such as Alpaca. For this purpose, we will assume access is available to the LLaMa files. 3. **List and install the programs required for LLaMa by running the following command from within the cloned repository directory:** `pip install -r requirements.txt pip install -e . ` 4. **Set up inference script:** The `example.py` script provided in the LLaMA repository can be used to run LLaMA inference. The script can be run on a single- or multi-gpu node with `torchrun` and will output completions for two pre-defined prompts. Open `example.py` and set the following parameters based on your preference. Descriptions for each parameter and what they do are listed below. - `--max_seq_len`: maximum sequence length (default is 2048) - `--max_batch_size`: maximum batch size (default is 8) - `--gen_length`: generation length (default is 512) - `--temperature`: generation temperature (default is 1.0) - `--top_p`: top-p sampling (default is 0.9) For example, to run LLaMA inference on a single GPU with checkpoint and tokenizer files in the directory `/path/to/files`, use the following command: `torchrun --nproc_per_node 1 example.py --ckpt_dir /path/to/files --tokenizer_path /path/to/files/tokenizer.model ` Note: The `nproc_per_node` argument depends on the size of the model and the number of GPUs required for the default LLaMa checkpoints. Refer to the following table to set the appropriate value for your system and model you are running: | Model | GPUs Required | | --- | --- | | 7B | 1 | | 13B | 2 | | 33B | 4 | | 65B | 8 | In the example below, we used the standard prompts in the example file but played with the `temperature` and `top_p` variables, yielding some interesting results. We also increased the response length. ![]() Be careful playing with the thermostat! ## Break Down Of The example.py File and Variables You Need To Know Most of these are preset within the model and do not need to be adjusted; however, if you are getting into using LLMs and inferencing, here is some handy information that breaks down what the flags and switches do. 1. `local_rank`: The rank of the current GPU (process) in the group of GPUs used for model parallelism. This is set automatically. 2. `world_size`: The total number of GPUs being used for model parallelism. This is set automatically. 3. `ckpt_dir`: The directory containing the model checkpoints. This is set at runtime. 4. `tokenizer_path`: The path to the tokenizer used to preprocess the text. This is set at runtime. 5. `temperature`: A parameter that controls the randomness of the generated text (higher values make the output more random, and lower values make it more focused). 6. `top_p`: A parameter that controls the nucleus sampling strategy, which is a method for generating text that selects only the most probable tokens with a cumulative probability of at most `top_p`. 7. `max_seq_len`: The maximum sequence length for the input and output of the model. You can adjust this to decrease your VRAM need. 8. `max_batch_size`: The maximum number of input sequences that the model can process in parallel. You can adjust this to decrease your VRAM need. 9. `prompts`: A list of input prompts for the model to generate text based on. 10. `results`: A list of generated text corresponding to each input prompt. ## Final Thoughts Our tests showed that the RTX 8000 GPUs could deliver impressive performance, with inference times ranging from a few seconds to about a minute, depending on the model parameters and input size. The GPUs could handle large batch sizes and run inference on multiple models simultaneously without any noticeable slowdown. One of the benefits of using GPUs for inference is that they can be easily scaled up by adding more GPUs to the system. This allows for even larger models to be run in real-time, enabling new applications in areas like chatbots, question-answering systems, and sentiment analysis. Alpaca is a compact AI language model created by a team of computer scientists at Stanford University. It’s built on Meta’s LLaMA 7B model, which boasts 7 billion parameters and has been trained using a vast amount of text from web. To fine-tune Alpaca, 52,000 instruction-following demonstrations were generated with OpenAI’s text-davinci-003 model, a versatile model capable of executing various tasks based on natural language instructions. Alpaca is better at following instructions and producing text for a range of purposes, including writing summaries, stories, and jokes. With a training cost under $600, Alpaca is designed to be both cost-effective and easily replicable. Highly impressively, it is worth mentioning that Alpaca can be run on even just a laptop. We are training a model similar to Alpaca using the methods that that Stanford did, but with our own twist. This is just one way we are working with the latest technological advancements in the lab, and thanks to some powerful NVIDIA GPUs, we can do even more than just inference; we are training. Be on the lookout for the next article soon about fine-tuning models like Meta LLaMa. **Engage with StorageReview** [Newsletter](https://www.storagereview.com/storage_newsletter) |  [YouTube](https://www.youtube.com/user/StorageReview "Opens in a new window") | Podcast  [iTunes](https://podcasts.apple.com/gb/podcast/storagereview-com-storage-reviews/id1060681115 "Opens in a new window")/ [Spotify](https://open.spotify.com/show/1y6VnznABhHeOSMOmbDTz0 "Opens in a new window") |  [Instagram](https://www.instagram.com/storagereview/ "Opens in a new window") |  [Twitter](https://twitter.com/storagereview "Opens in a new window") |  [TikTok](https://www.tiktok.com/@storagereview? "Opens in a new window") |  [RSS Feed](https://www.storagereview.com/rss.xml) ![]() ### Jordan Ranous AI Specialist; navigating you through the world of Enterprise AI. Writer and Analyst for Storage Review, coming from a background of Financial Big Data Analytics, Datacenter Ops/DevOps, and CX Analytics. Pilot, Astrophotographer, LTO Tape Guru, and Battery/Solar Enthusiast. Previous post: [StorageReview Calculated 100 Trillion Digits of Pi in 54 days, Besting Google Cloud](https://www.storagereview.com/review/storagereview-calculated-100-trillion-digits-of-pi-in-54-days-besting-google-cloud) Next post: [HPE Alletra MP Hardware Deep Dive](https://www.storagereview.com/review/hpe-alletra-mp-hardware-deep-dive) Trusted Vendors Products and solutions from our affiliate partners: - [ ![Ubiquiti Logo](https://www.storagereview.com/wp-content/uploads/2024/11/2-1.webp) Ubiquiti ](https://store.ui.com/us/en?a_aid=StorageReview "Opens in a new window") - [ ![Newegg Logo](https://www.storagereview.com/wp-content/uploads/2025/11/newegg.webp) Newegg ](https://click.linksynergy.com/fs-bin/click?id=g5terNMDhz0&offerid=1207190.18&subid=0&type=4 "Opens in a new window") - [ ![Amazon Logo](https://www.storagereview.com/wp-content/uploads/2025/11/amazon.webp) Amazon ](https://amzn.to/4fLJUEW "Opens in a new window") Newsletter Subscribe to the StorageReview newsletter to stay up to date on the latest news and reviews. We promise no spam! 1 Leave this field empty if you’re human: ## Advertisement Content Categories [Facebook](https://www.facebook.com/@storagereview) [X](https://twitter.com/storagereview) [Instagram](https://www.instagram.com/storagereview/) [LinkedIn](https://www.linkedin.com/company/storagereview-com) [YouTube](https://www.youtube.com/user/storagereview?sub_confirmation=1) [Email](mailto:info@storagereview.com) [Spotify](https://open.spotify.com/show/1y6VnznABhHeOSMOmbDTz0) [Reddit](https://www.reddit.com/r/StorageReview/) [Discord](https://discord.gg/TwMHb4azdC) [RSS](https://www.storagereview.com/rss.xml) [TikTok](https://www.tiktok.com/@storagereview) Copyright © 1998-2025 Flying Pig Ventures, LLC Cincinnati, Ohio. All rights reserved. Manage your privacy To provide the best experiences, we and our partners use technologies like cookies to store and/or access device information. Consenting to these technologies will allow us and our partners to process personal data such as browsing behavior or unique IDs on this site and show (non-) personalized ads. Not consenting or withdrawing consent, may adversely affect certain features and functions. Click below to consent to the above or make granular choices. Your choices will be applied to this site only. You can change your settings at any time, including withdrawing your consent, by using the toggles on the Cookie Policy, or by clicking on the manage consent button at the bottom of the screen. Functional Functional Always active The technical storage or access is strictly necessary for the legitimate purpose of enabling the use of a specific service explicitly requested by the subscriber or user, or for the sole purpose of carrying out the transmission of a communication over an electronic communications network. Preferences Preferences The technical storage or access is necessary for the legitimate purpose of storing preferences that are not requested by the subscriber or user. Statistics Statistics The technical storage or access that is used exclusively for statistical purposes. The technical storage or access that is used exclusively for anonymous statistical purposes. Without a subpoena, voluntary compliance on the part of your Internet Service Provider, or additional records from a third party, information stored or retrieved for this purpose alone cannot usually be used to identify you. Marketing Marketing The technical storage or access is required to create user profiles to send advertising, or to track the user on a website or across several websites for similar marketing purposes. Statistics Marketing Features Always active Special features Special purposes Always active - Manage options - Manage services - Manage {vendor\_count} vendors - [Read more about these purposes](https://cookiedatabase.org/tcf/purposes/) Accept Deny Manage options Save preferences Manage options - {title} - {title} - {title} Manage your privacy To provide the best experiences, we use technologies like cookies to store and/or access device information. Consenting to these technologies will allow us to process data such as browsing behavior or unique IDs on this site. Not consenting or withdrawing consent, may adversely affect certain features and functions. Functional Functional Always active The technical storage or access is strictly necessary for the legitimate purpose of enabling the use of a specific service explicitly requested by the subscriber or user, or for the sole purpose of carrying out the transmission of a communication over an electronic communications network. Preferences Preferences The technical storage or access is necessary for the legitimate purpose of storing preferences that are not requested by the subscriber or user. Statistics Statistics The technical storage or access that is used exclusively for statistical purposes. The technical storage or access that is used exclusively for anonymous statistical purposes. Without a subpoena, voluntary compliance on the part of your Internet Service Provider, or additional records from a third party, information stored or retrieved for this purpose alone cannot usually be used to identify you. Marketing Marketing The technical storage or access is required to create user profiles to send advertising, or to track the user on a website or across several websites for similar marketing purposes. Statistics Marketing Features Always active Special features Special purposes Always active - Manage options - Manage services - Manage {vendor\_count} vendors - [Read more about these purposes](https://cookiedatabase.org/tcf/purposes/) Accept Deny Manage options Save preferences Manage options - {title} - {title} - {title} Manage consent Manage consent ```json {"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/www.storagereview.com\/review\/meta-llama-and-alpacas-loose-in-the-lab-running-large-language-models-locally","url":"https:\/\/www.storagereview.com\/review\/meta-llama-and-alpacas-loose-in-the-lab-running-large-language-models-locally","name":"Meta LLaMa And Alpacas Loose in the Lab! Running Large Language Models Locally - StorageReview.com","isPartOf":{"@id":"https:\/\/www.storagereview.com\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.storagereview.com\/review\/meta-llama-and-alpacas-loose-in-the-lab-running-large-language-models-locally#primaryimage"},"image":{"@id":"https:\/\/www.storagereview.com\/review\/meta-llama-and-alpacas-loose-in-the-lab-running-large-language-models-locally#primaryimage"},"thumbnailUrl":"https:\/\/www.storagereview.com\/wp-content\/uploads\/2023\/04\/dual-rtx8000.jpg","datePublished":"2023-04-19T20:25:16+00:00","dateModified":"2023-06-16T15:38:07+00:00","description":"Large language models like GPT-4, meta LLaMa, and Alpaca pushing the boundaries of natural language processing and hardware to run them.","breadcrumb":{"@id":"https:\/\/www.storagereview.com\/review\/meta-llama-and-alpacas-loose-in-the-lab-running-large-language-models-locally#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.storagereview.com\/review\/meta-llama-and-alpacas-loose-in-the-lab-running-large-language-models-locally"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.storagereview.com\/review\/meta-llama-and-alpacas-loose-in-the-lab-running-large-language-models-locally#primaryimage","url":"https:\/\/www.storagereview.com\/wp-content\/uploads\/2023\/04\/dual-rtx8000.jpg","contentUrl":"https:\/\/www.storagereview.com\/wp-content\/uploads\/2023\/04\/dual-rtx8000.jpg","width":1920,"height":1440},{"@type":"BreadcrumbList","@id":"https:\/\/www.storagereview.com\/review\/meta-llama-and-alpacas-loose-in-the-lab-running-large-language-models-locally#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.storagereview.com\/"},{"@type":"ListItem","position":2,"name":"Review","item":"https:\/\/www.storagereview.com\/review"},{"@type":"ListItem","position":3,"name":"Meta LLaMa And Alpacas Loose in the Lab! Running Large Language Models Locally"}]},{"@type":"WebSite","@id":"https:\/\/www.storagereview.com\/#website","url":"https:\/\/www.storagereview.com\/","name":"StorageReview.com","description":"StorageReview.com is a leading provider of news and reviews throughout the entire IT stack - from the datacenter to the edge, and all points in between.","publisher":{"@id":"https:\/\/www.storagereview.com\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.storagereview.com\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.storagereview.com\/#organization","name":"StorageReview.com","url":"https:\/\/www.storagereview.com\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.storagereview.com\/#\/schema\/logo\/image\/","url":"https:\/\/www.storagereview.com\/wp-content\/uploads\/2020\/02\/Storage-Reviews-2-2.png","contentUrl":"https:\/\/www.storagereview.com\/wp-content\/uploads\/2020\/02\/Storage-Reviews-2-2.png","width":344,"height":61,"caption":"StorageReview.com"},"image":{"@id":"https:\/\/www.storagereview.com\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/x.com\/storagereview","http:\/\/youtube.com\/user\/storagereview"]}]} ```